Abstract
In addition to asking questions about facts in the world, some internet users—in particular, second language learners—ask questions about language itself. Depending on their proficiency level and audience, they may pose these questions in an L1 (first language) or an L2 (second language). We investigate how multilingual LLMs perform at crosslingual metalinguistic question answering. Focusing on binary questions about sentence grammaticality constructed from error-annotated learner corpora, we prompt three LLMs (Aya, Llama, and GPT) in multiple languages, including English, German, Korean, Russian, and Ukrainian. Our study reveals that the language of the prompt can significantly affect model performance, and despite English being the dominant training language for all three models, prompting in a different language with questions about English often yields better results.- Anthology ID:
- 2024.findings-emnlp.916
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2024
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15622–15634
- Language:
- URL:
- https://aclanthology.org/2024.findings-emnlp.916
- DOI:
- 10.18653/v1/2024.findings-emnlp.916
- Cite (ACL):
- Shabnam Behzad, Amir Zeldes, and Nathan Schneider. 2024. To Ask LLMs about English Grammaticality, Prompt Them in a Different Language. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15622–15634, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- To Ask LLMs about English Grammaticality, Prompt Them in a Different Language (Behzad et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2024.findings-emnlp.916.pdf