@inproceedings{cai-etal-2025-mhalo,
    title = "{MHALO}: Evaluating {MLLM}s as Fine-grained Hallucination Detectors",
    author = "Cai, Yishuo  and
      Gu, Renjie  and
      Li, Jiaxu  and
      Huang, Xuancheng  and
      Chen, Junzhe  and
      Gu, Xiaotao  and
      Huang, Minlie",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.478/",
    pages = "9197--9222",
    ISBN = "979-8-89176-256-5",
    abstract = "Hallucination remains a critical challenge for multimodal large language models (MLLMs), undermining their reliability in real-world applications. While fine-grained hallucination detection (FHD) holds promise for enhancing high-quality vision-language data construction and model alignment through enriched feedback signals, automated solutions for this task have yet to be systematically explored. Inspired by the concept of ``MLLM as a Judge'', we introduce MHALO, the first comprehensive benchmark specifically designed for evaluating MLLMs' capability in performing token-level FHD. Our benchmark encompasses 12 distinct hallucination types spanning both multimodal perception and reasoning domains. Through extensive evaluations of 9 selected MLLMs, we reveal substantial performance limitations, with the leading model achieving an average $F1_{IoU}$ of only 40.59{\%}. To address this limitation, we develop HaloDet-4B, a specialized model trained on our curated training data, which significantly outperforms existing models. We hope the benchmark can provide valuable insights for future research on hallucination mitigation in MLLMs. The code and dataset will be publicly available."
}