@article{mannion-etal-2026-biomedical,
    title = "Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New {F}rench Health Corpus",
    author = "Mannion, Aidan  and
      Macaire, C{\'e}cile  and
      Violle, Armand  and
      Ohayon, St{\'e}phane  and
      Tannier, Xavier  and
      Schwab, Didier  and
      Goeuriot, Lorraine  and
      Portet, Fran{\c{c}}ois",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    journal = "International Conference on Language Resources and Evaluation",
    volume = "main",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.lrec-main.833/",
    pages = "10617--10633",
    abstract = "Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, yet their adaptation to specialized fields remains challenging, particularly for non-English languages. This study investigates domain-adaptive pre-training (DAPT) as a strategy for specializing small to mid-sized LLMs in the French biomedical domain through continued pre-training. We address two key research questions: the viability of specialized continued pre-training for domain adaptation and the relationship between domain-specific performance gains and general capability degradation. Our contributions include the release of a fully open-licensed French biomedical corpus suitable for commercial and open-source applications, the training and release of specialized French biomedical LLMs, and novel insights for DAPT implementation. Our methodology encompasses the collection and refinement of high-quality French biomedical texts, the exploration of causal language modeling approaches using DAPT, and conducting extensive comparative evaluations. Our results cast doubt on the efficacy of DAPT, in contrast to previous works, but we highlight its viability in smaller-scale, resource-constrained scenarios under the right conditions. Our findings further suggest that model merging post-DAPT is essential to mitigate generalization trade-offs, and in some cases even improves performance on specialized tasks at which the DAPT was directed."
}