Abstract
Recent psycholinguistic theories emphasize the interdependence between linguistic expectations and memory limitations in human language processing. We modify the self-attention mechanism of a transformer model to simulate a lossy context representation, biasing the model’s predictions to give additional weight to the local linguistic context. We show that surprisal estimates from our locally-biased model generally provide a better fit to human psychometric data, underscoring the sensitivity of the human parser to local linguistic information.- Anthology ID:
- 2024.cmcl-1.3
- Volume:
- Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Tatsuki Kuribayashi, Giulia Rambelli, Ece Takmaz, Philipp Wicke, Yohei Oseki
- Venues:
- CMCL | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 30–36
- Language:
- URL:
- https://aclanthology.org/2024.cmcl-1.3
- DOI:
- Cite (ACL):
- Andrea De Varda and Marco Marelli. 2024. Locally Biased Transformers Better Align with Human Reading Times. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 30–36, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Locally Biased Transformers Better Align with Human Reading Times (De Varda & Marelli, CMCL-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.cmcl-1.3.pdf