@inproceedings{nainia-etal-2025-beyond,
    title = "Beyond {BLEU}: Ethical Risks of Misleading Evaluation in Domain-Specific {QA} with {LLM}s",
    author = "Nainia, Ayoub  and
      Vignes-Lebbe, R{\'e}gine  and
      Mousannif, Hajar  and
      Zahir, Jihad",
    editor = "Picazo-Izquierdo, Alicia  and
      Estevanell-Valladares, Ernesto Luis  and
      Mitkov, Ruslan  and
      Guillena, Rafael Mu{\~n}oz  and
      Cerd{\'a}, Ra{\'u}l Garc{\'i}a",
    booktitle = "Proceedings of the First Workshop on Comparative Performance Evaluation: From Rules to Language Models",
    month = sep,
    year = "2025",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd., Shoumen, Bulgaria",
    url = "https://preview.aclanthology.org/corrections-2026-01/2025.r2lm-1.9/",
    pages = "77--86",
    abstract = "Large Language Models (LLMs) are increasingly used in scientific question answering (QA), including high-stakes fields such as biodiversity informatics. However, standard evaluation metrics such as BLEU, ROUGE, Exact Match (EM), and BERTScore remain poorly aligned with the factual and domain-specific requirements of these tasks. In this work, we investigate the gap between automatic metrics and expert judgment in botanical QA by comparing metric scores with human ratings across five dimensions: accuracy, completeness, relevance, fluency, and terminology usage. Our results show that standard metrics often misrepresent response quality, particularly in the presence of paraphrasing, omission, or domain-specific language. Through both quantitative analysis and qualitative examples, we show that high-scoring responses may still exhibit critical factual errors or omissions. These findings highlight the need for domain-aware evaluation frameworks that incorporate expert feedback and raise important ethical concerns about the deployment of LLMs in scientific contexts."
}