@inproceedings{sanchez-carmona-etal-2024-multilevel,
    title = "A Multilevel Analysis of {P}ub{M}ed-only {BERT}-based Biomedical Models",
    author = "Sanchez Carmona, Vicente  and
      Jiang, Shanshan  and
      Dong, Bin",
    editor = "Naumann, Tristan  and
      Ben Abacha, Asma  and
      Bethard, Steven  and
      Roberts, Kirk  and
      Bitterman, Danielle",
    booktitle = "Proceedings of the 6th Clinical Natural Language Processing Workshop",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.clinicalnlp-1.10/",
    doi = "10.18653/v1/2024.clinicalnlp-1.10",
    pages = "105--110",
    abstract = "Biomedical NLP models play a big role in the automatic extraction of information from biomedical documents, such as COVID research papers. Three landmark models have led the way in this area: BioBERT, MSR BiomedBERT, and BioLinkBERT. However, their shallow evaluation {--}a single mean score{--} forbid us to better understand how the contributions proposed in each model advance the Biomedical NLP field. We show through a Multilevel Analysis how we can assess these contributions. Our analyses across 5000 fine-tuned models show that, actually, BiomedBERT`s true effect is bigger than BioLinkBERT`s effect, and the success of BioLinkBERT does not seem to be due to its contribution {--}the Link function{--} but due to an unknown factor."
}