@inproceedings{maharaj-etal-2025-etf,
    title = "{ETF}: An Entity Tracing Framework for Hallucination Detection in Code Summaries",
    author = "Maharaj, Kishan  and
      Munigala, Vitobha  and
      Tamilselvam, Srikanth G.  and
      Kumar, Prince  and
      Sen, Sayandeep  and
      Kodeswaran, Palani  and
      Mishra, Abhijit  and
      Bhattacharyya, Pushpak",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1480/",
    pages = "30639--30652",
    ISBN = "979-8-89176-251-0",
    abstract = "Recent advancements in large language models (LLMs) have significantly enhanced their ability to understand both natural language and code, driving their use in tasks like natural language-to-code (NL2Code) and code summarisation. However, LLMs are prone to hallucination{---}outputs that stray from intended meanings. Detecting hallucinations in code summarisation is especially difficult due to the complex interplay between programming and natural languages. We introduce a first-of-its-kind dataset, CodeSumEval, with {\textasciitilde}10K samples, curated specifically for hallucination detection in code summarisation. We further propose a novel Entity Tracing Framework (ETF) that a) utilises static program analysis to identify code entities from the program and b) uses LLMs to map and verify these entities and their intents within generated code summaries. Our experimental analysis demonstrates the framework{'}s effectiveness, leading to a 73{\%} F1 score. The proposed approach provides a method for detecting hallucinations by tracing entities from the summary to the code, allowing us to evaluate summary accuracy and localise the error within the summary."
}