The Effects of Surprisal across Languages: Results from Native and Non-native Reading

Andrea de Varda, Marco Marelli


Abstract
It is well known that the surprisal of an upcoming word, as estimated by language models, is a solid predictor of reading times (Smith and Levy, 2013). However, most of the studies that support this view are based on English and few other Germanic languages, leaving an open question as to the cross-lingual generalizability of such findings. Moreover, they tend to consider only the best-performing eye-tracking measure, which might conflate the effects of predictive and integrative processing. Furthermore, it is not clear whether prediction plays a role in non-native language processing in bilingual individuals (Grüter et al., 2014). We approach these problems at large scale, extracting surprisal estimates from mBERT, and assessing their psychometric predictive power on the MECO corpus, a cross-linguistic dataset of eye movement behavior in reading (Siegelman et al., 2022; Kuperman et al., 2020). We show that surprisal is a strong predictor of reading times across languages and fixation measurements, and that its effects in L2 are weaker with respect to L1.
Anthology ID:
2022.findings-aacl.13
Volume:
Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022
Month:
November
Year:
2022
Address:
Online only
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
138–144
Language:
URL:
https://aclanthology.org/2022.findings-aacl.13
DOI:
Bibkey:
Cite (ACL):
Andrea de Varda and Marco Marelli. 2022. The Effects of Surprisal across Languages: Results from Native and Non-native Reading. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 138–144, Online only. Association for Computational Linguistics.
Cite (Informal):
The Effects of Surprisal across Languages: Results from Native and Non-native Reading (de Varda & Marelli, Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.findings-aacl.13.pdf