Get the Best out of 1B LLMs: Insights from Information Extraction on Clinical Documents
Saeed Farzi, Soumitra Ghosh, Alberto Lavelli, Bernardo Magnini
Abstract
While the popularity of large, versatile language models like ChatGPT continues to rise, the landscape shifts when considering open-source models tailored to specific domains. Moreover, many areas, such as clinical documents, suffer from a scarcity of training data, often amounting to only a few hundred instances. Additionally, in certain settings, such as hospitals, cloud-based solutions pose privacy concerns, necessitating the deployment of language models on traditional hardware, such as single GPUs or powerful CPUs. To address these complexities, we conduct extensive experiments on both clinical entity detection and relation extraction in clinical documents using 1B parameter models. Our study delves into traditional fine-tuning, continuous pre-training in the medical domain, and instruction-tuning methods, providing valuable insights into their effectiveness in a multilingual setting. Our results underscore the importance of domain-specific models and pre-training for clinical natural language processing tasks. Furthermore, data augmentation using cross-lingual information improves performance in most cases, highlighting the potential for multilingual enhancements.- Anthology ID:
- 2024.bionlp-1.21
- Volume:
- Proceedings of the 23rd Workshop on Biomedical Natural Language Processing
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Dina Demner-Fushman, Sophia Ananiadou, Makoto Miwa, Kirk Roberts, Junichi Tsujii
- Venues:
- BioNLP | WS
- SIG:
- SIGBIOMED
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 266–276
- Language:
- URL:
- https://aclanthology.org/2024.bionlp-1.21
- DOI:
- Cite (ACL):
- Saeed Farzi, Soumitra Ghosh, Alberto Lavelli, and Bernardo Magnini. 2024. Get the Best out of 1B LLMs: Insights from Information Extraction on Clinical Documents. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 266–276, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Get the Best out of 1B LLMs: Insights from Information Extraction on Clinical Documents (Farzi et al., BioNLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.bionlp-1.21.pdf