Abstract
Eye movements in reading reveal humans’ cognitive processes involved in language understanding. The duration a reader’s eyes fixate on a word has been used as a measure of the visual attention given to that word or its significance to the reader. This study investigates the correlation between the importance attributed to input tokens by language models (LMs) on the one hand and humans, in the form of fixation durations, on the other hand. While previous research on the internal processes of LMs have employed the models’ attention weights, recent studies have argued in favor of gradient-based methods. Moreover, previous approaches to interpret LMs’ internals with human gaze have neglected the tasks readers performed during reading, even though psycholinguistic research underlines that reading patterns are task-dependent. We therefore employ a gradient-based saliency method to measure the importance of input tokens when LMs are targeted on specific tasks, and we find that task specificity plays a crucial role in the correlation between human- and model-assigned importance. Our implementation is available at https://github.com/gjwubyron/Scan.- Anthology ID:
- 2024.cmcl-1.22
- Volume:
- Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Tatsuki Kuribayashi, Giulia Rambelli, Ece Takmaz, Philipp Wicke, Yohei Oseki
- Venues:
- CMCL | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 255–263
- Language:
- URL:
- https://aclanthology.org/2024.cmcl-1.22
- DOI:
- Cite (ACL):
- Guojun Wu, Lena Bolliger, David Reich, and Lena Jäger. 2024. An Eye Opener Regarding Task-Based Text Gradient Saliency. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 255–263, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- An Eye Opener Regarding Task-Based Text Gradient Saliency (Wu et al., CMCL-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.cmcl-1.22.pdf