@inproceedings{riina-etal-2024-evaluation,
    title = "An Evaluation of {E}nglish to {S}panish Medical Translation by Large Language Models",
    author = "Riina, Nicholas  and
      Patlolla, Likhitha  and
      Hernandez Joya, Camilo  and
      Bautista, Roger  and
      Olivar-Villanueva, Melissa  and
      Kumar, Anish",
    editor = "Martindale, Marianna  and
      Campbell, Janice  and
      Savenkov, Konstantin  and
      Goel, Shivali",
    booktitle = "Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations)",
    month = sep,
    year = "2024",
    address = "Chicago, USA",
    publisher = "Association for Machine Translation in the Americas",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.amta-presentations.15/",
    pages = "222--236",
    abstract = "Machine translation (MT) with Large Language Models (LLMs) holds promise as a clinical translation tool with more capabilities than a traditional MT model. This work compares the quality of English to Spanish translation by three LLMs: ChatGPT3.5 Turbo, ChatGPT4o, and Aguila, against Google Translate. The test set used in this study is MedlinePlus, a parallel dataset of educational health information in English and Spanish developed by the National Library of Medicine. ChatGPT4o and Google Translate performed similarly in both automated scoring (BLEU, METEOR, and BERTscore) and human evaluation with ChatGPT3.5 Turbo not far behind. Aguila, the only LLM intended for primarily Spanish and Catalan use, surprisingly performed much worse than the other models. However, qualitative analysis of Aguila{'}s results revealed the use of Spanish word choice that may reach a broader audience."
}