@inproceedings{bao-etal-2025-faithbench,
    title = "{F}aith{B}ench: A Diverse Hallucination Benchmark for Summarization by {M}odern {LLM}s",
    author = "Bao, Forrest Sheng  and
      Li, Miaoran  and
      Qu, Renyi  and
      Luo, Ge  and
      Wan, Erana  and
      Tang, Yujia  and
      Fan, Weisi  and
      Tamber, Manveer Singh  and
      Kazi, Suleman  and
      Sourabh, Vivek  and
      Qi, Mike  and
      Tu, Ruixuan  and
      Xu, Chenyu  and
      Gonzales, Matthew  and
      Mendelevitch, Ofer  and
      Ahmad, Amin",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-short.38/",
    pages = "448--461",
    ISBN = "979-8-89176-190-2",
    abstract = "Summarization is one of the most common tasks performed by large language models (LLMs), especially in applications like Retrieval-Augmented Generation (RAG). However, existing evaluations of hallucinations in LLM-generated summaries, and evaluations of hallucination detection models both suffer from a lack of diversity and recency in the LLM and LLM families considered. This paper introduces FaithBench, a summarization hallucination benchmark comprising challenging hallucinations made by 10 modern LLMs from 8 different families, with ground truth annotations by human experts. ``Challenging'' here means summaries on which popular, state-of-the-art hallucination detection models, including GPT-4o-as-a-judge, disagreed on. Our results show GPT-4o and GPT-3.5-Turbo produce the least hallucinations. However, most state-of-the-art hallucination detection models have near 50{\%} accuracies on FaithBench, indicating lots of room for future improvement."
}