Abstract
This paper investigates whether adding data of typologically closer languages improves the performance of transformer-based models for three different downstream tasks, namely Part-of-Speech tagging, Named Entity Recognition, and Sentiment Analysis, compared to a monolingual and plain multilingual language model. For the presented pilot study, we performed experiments for the use case of Slovene, a low(er)-resourced language belonging to the Slavic language family. The experiments were carried out in a controlled setting, where a monolingual model for Slovene was compared to combined language models containing Slovene, trained with the same amount of Slovene data. The experimental results show that adding typologically closer languages indeed improves the performance of the Slovene language model, and even succeeds in outperforming the large multilingual XLM-RoBERTa model for NER and PoS-tagging. We also reveal that, contrary to intuition, distantly or unrelated languages also combine admirably with Slovene, often out-performing XLM-R as well. All the bilingual models used in the experiments are publicly available at https://github.com/pranaydeeps/BLAIR- Anthology ID:
- 2023.bsnlp-1.5
- Volume:
- Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023)
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Jakub Piskorski, Michał Marcińczuk, Preslav Nakov, Maciej Ogrodniczuk, Senja Pollak, Pavel Přibáň, Piotr Rybak, Josef Steinberger, Roman Yangarber
- Venue:
- BSNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 32–39
- Language:
- URL:
- https://aclanthology.org/2023.bsnlp-1.5
- DOI:
- 10.18653/v1/2023.bsnlp-1.5
- Cite (ACL):
- Pranaydeep Singh, Aaron Maladry, and Els Lefever. 2023. Too Many Cooks Spoil the Model: Are Bilingual Models for Slovene Better than a Large Multilingual Model?. In Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023), pages 32–39, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Too Many Cooks Spoil the Model: Are Bilingual Models for Slovene Better than a Large Multilingual Model? (Singh et al., BSNLP 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.bsnlp-1.5.pdf