Transformer Attention vs Human Attention in Anaphora Resolution
Anastasia Kozlova, Albina Akhmetgareeva, Aigul Khanova, Semen Kudriavtsev, Alena Fenogenova
Abstract
Motivated by human cognitive processes, attention mechanism within transformer architecture has been developed to assist neural networks in allocating focus to specific aspects within input data. Despite claims regarding the interpretability achieved by attention mechanisms, the extent of correlation and similarity between machine and human attention remains a subject requiring further investigation.In this paper, we conduct a quantitative analysis of human attention compared to neural attention mechanisms in the context of the anaphora resolution task. We collect an eye-tracking dataset based on the Winograd schema challenge task for the Russian language. Leveraging this dataset, we conduct an extensive analysis of the correlations between human and machine attention maps across various transformer architectures, network layers of pre-trained and fine-tuned models. Our aim is to investigate whether insights from human attention mechanisms can be used to enhance the performance of neural networks in tasks such as anaphora resolution. The results reveal distinctions in anaphora resolution processing, offering promising prospects for improving the performance of neural networks and understanding the cognitive nuances of human perception.- Anthology ID:
- 2024.cmcl-1.10
- Volume:
- Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Tatsuki Kuribayashi, Giulia Rambelli, Ece Takmaz, Philipp Wicke, Yohei Oseki
- Venues:
- CMCL | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 109–122
- Language:
- URL:
- https://aclanthology.org/2024.cmcl-1.10
- DOI:
- Cite (ACL):
- Anastasia Kozlova, Albina Akhmetgareeva, Aigul Khanova, Semen Kudriavtsev, and Alena Fenogenova. 2024. Transformer Attention vs Human Attention in Anaphora Resolution. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 109–122, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Transformer Attention vs Human Attention in Anaphora Resolution (Kozlova et al., CMCL-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.cmcl-1.10.pdf