@inproceedings{schmidtova-etal-2025-eyes,
    title = "Do My Eyes Deceive Me? A Survey of Human Evaluations of Hallucinations in {NLG}",
    author = "Schmidtova, Patricia  and
      Cal{\`o}, Eduardo  and
      Balloccu, Simone  and
      Gkatzia, Dimitra  and
      Huidrom, Rudali  and
      Lango, Mateusz  and
      Same, Fahime  and
      Zouhar, Vil{\'e}m  and
      Mahamood, Saad  and
      Dusek, Ondrej",
    editor = "Flek, Lucie  and
      Narayan, Shashi  and
      Phương, L{\^e} Hồng  and
      Pei, Jiahuan",
    booktitle = "Proceedings of the 18th International Natural Language Generation Conference",
    month = oct,
    year = "2025",
    address = "Hanoi, Vietnam",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-lei-gao-usc/2025.inlg-main.4/",
    pages = "60--79",
    abstract = "Hallucinations are one of the most pressing challenges for large language models (LLMs). While numerous methods have been proposed to detect and mitigate them automatically, human evaluation continues to serve as the gold standard. However, these human evaluations of hallucinations show substantial variation in definitions, terminology, and evaluation practices. In this paper, we survey 64 studies involving human evaluation of hallucination published between 2019 and 2024, to investigate how hallucinations are currently defined and assessed. Our analysis reveals a lack of consistency in definitions and exposes several concerning methodological shortcomings. Crucial details, such as evaluation guidelines, user interface design, inter-annotator agreement metrics, and annotator demographics, are frequently under-reported or omitted altogether."
}