Eyes Show the Way: Modelling Gaze Behaviour for Hallucination Detection

Kishan Maharaj, Ashita Saxena, Raja Kumar, Abhijit Mishra, Pushpak Bhattacharyya

Abstract
Detecting hallucinations in natural language processing (NLP) is a critical undertaking that demands a deep understanding of both the semantic and pragmatic aspects of languages. Cognitive approaches that leverage users’ behavioural signals, such as gaze, have demonstrated effectiveness in addressing NLP tasks with similar linguistic complexities. However, their potential in the context of hallucination detection remains largely unexplored. In this paper, we propose a novel cognitive approach for hallucination detection that leverages gaze signals from humans. We first collect and introduce an eye tracking corpus (IITB-HGC: IITB-Hallucination Gaze corpus) consisting of 500 instances, annotated by five annotators for hallucination detection. Our analysis reveals that humans selectively attend to relevant parts of the text based on distributional similarity, similar to the attention bias phenomenon in psychology. We identify two attention strategies employed by humans: global attention, which focuses on the most informative sentence, and local attention, which focuses on important words within a sentence. Leveraging these insights, we propose a novel cognitive framework for hallucination detection that incorporates these attention biases. Experimental evaluations on the FactCC dataset demonstrate the efficacy of our approach, obtaining a balanced accuracy of 87.1%. Our study highlights the potential of gaze-based approaches in addressing the task of hallucination detection and sheds light on the cognitive processes employed by humans in identifying inconsistencies.
Anthology ID:
2023.findings-emnlp.764
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
11424–11438
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.764
DOI:
10.18653/v1/2023.findings-emnlp.764
Bibkey:
Cite (ACL):
Kishan Maharaj, Ashita Saxena, Raja Kumar, Abhijit Mishra, and Pushpak Bhattacharyya. 2023. Eyes Show the Way: Modelling Gaze Behaviour for Hallucination Detection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11424–11438, Singapore. Association for Computational Linguistics.
Cite (Informal):
Eyes Show the Way: Modelling Gaze Behaviour for Hallucination Detection (Maharaj et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2023.findings-emnlp.764.pdf
PDF Search