@inproceedings{buhnila-etal-2026-tracklist,
    title = "{T}rack{L}ist: Tracing Back Query Linguistic Diversity for Head and Tail Medical Knowledge in Open Large Language Models",
    author = "Buhnila, Ioana  and
      Sinha, Aman  and
      Constant, Mathieu",
    editor = "Demner-Fushman, Dina  and
      Ananiadou, Sophia  and
      Roberts, Kirk  and
      Tsujii, Junichi",
    booktitle = "{B}io{NLP} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.bionlp-1.33/",
    pages = "413--425",
    ISBN = "979-8-89176-434-7",
    abstract = "While humans can easily produce various types of answers, such as definitions, examples or paraphrases, Large Language Models (LLMs) struggle to provide correct answers to medical questions that require diverse answer formats. In this paper, we introduce TrackList, a fine-grained linguistic and statistical analysis pipeline to investigate the impact of the pre-training data on LLMs answers to diverse linguistic queries. We also propose RefoMed-EN, a medical dataset consisting of 6,170 human-annotated medical terms alongside their corresponding definitions, denominations, exemplifications, explanations, or paraphrases. We investigated whether the high or low frequency of a concept (head or tail knowledge) impacts the language model{'}s performance for answering medical questions. We evaluated the quality of the LLM{'}s output using syntactic and semantic similarity metrics, statistical correlations and embeddings. Results showed that the LLM{'}s answer quality for definition-type questions is the highest, while for the exemplification-type being the lowest. Additionally, we showed that for definition-type medical questions ({''}What is multiple sclerosis?''), LLMs are prone to paraphrase more for popular medical concepts, and less on more specialized medical knowledge."
}