@inproceedings{christophe-etal-2024-beyond,
    title = "Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical {LLM}s.",
    author = "Christophe, Clement  and
      Raha, Tathagata  and
      Maslenkova, Svetlana  and
      Salman, Muhammad Umar  and
      Kanithi, Praveenkumar  and
      Pimentel, Marco AF  and
      Khan, Shadab",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-emnlp.618/",
    doi = "10.18653/v1/2024.findings-emnlp.618",
    pages = "10549--10561",
    abstract = "Large Language Models (LLMs) have demonstrated significant potential in revolutionizing clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals nuanced insights. While continuous pretraining beyond 250 billion tokens yields marginal improvements, instruct fine-tuning emerges as a more influential factor. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. These findings underscore the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain."
}