„Mann“ is to “Donna” as「国王」is to « Reine » Adapting the Analogy Task for Multilingual and Contextual Embeddings
Timothee Mickus, Eduardo Calò, Léo Jacqmin, Denis Paperno, Mathieu Constant
Abstract
How does the word analogy task fit in the modern NLP landscape? Given the rarity of comparable multilingual benchmarks and the lack of a consensual evaluation protocol for contextual models, this remains an open question. In this paper, we introduce MATS: a multilingual analogy dataset, covering forty analogical relations in six languages, and evaluate human as well as static and contextual embedding performances on the task. We find that not all analogical relations are equally straightforward for humans, static models remain competitive with contextual embeddings, and optimal settings vary across languages and analogical relations. Several key challenges remain, including creating benchmarks that align with human reasoning and understanding what drives differences across methodologies.- Anthology ID:
- 2023.starsem-1.25
- Volume:
- Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Alexis Palmer, Jose Camacho-collados
- Venue:
- *SEM
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 270–283
- Language:
- URL:
- https://aclanthology.org/2023.starsem-1.25
- DOI:
- 10.18653/v1/2023.starsem-1.25
- Cite (ACL):
- Timothee Mickus, Eduardo Calò, Léo Jacqmin, Denis Paperno, and Mathieu Constant. 2023. „Mann“ is to “Donna” as「国王」is to « Reine » Adapting the Analogy Task for Multilingual and Contextual Embeddings. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 270–283, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- „Mann“ is to “Donna” as「国王」is to « Reine » Adapting the Analogy Task for Multilingual and Contextual Embeddings (Mickus et al., *SEM 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.starsem-1.25.pdf