@inproceedings{simonds-etal-2024-modem,
    title = "{M}o{DEM}: Mixture of Domain Expert Models",
    author = "Simonds, Toby  and
      Kurniawan, Kemal  and
      Lau, Jey Han",
    editor = "Baldwin, Tim  and
      Rodr{\'i}guez M{\'e}ndez, Sergio Jos{\'e}  and
      Kuo, Nicholas",
    booktitle = "Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association",
    month = dec,
    year = "2024",
    address = "Canberra, Australia",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.alta-1.6/",
    pages = "75--88",
    abstract = "We propose a novel approach to enhancing the performance and efficiency of large language models (LLMs) by combining domain prompt routing with domain-specialized models. We introduce a system that utilizes a BERT-based router to direct incoming prompts to the most appropriate domain expert model. These expert models are specifically tuned for domains such as health, mathematics and science. Our research demonstrates that this approach can significantly outperform general-purpose models of comparable size, leading to a superior performance-to-cost ratio across various benchmarks. The implications of this study suggest a potential shift in LLM development and deployment. Rather than focusing solely on creating increasingly large, general-purpose models, the future of AI may lie in developing ecosystems of smaller, highly specialized models coupled with sophisticated routing systems. This approach could lead to more efficient resource utilization, reduced computational costs, and superior overall performance."
}