Assessment of Massively Multilingual Sentiment Classifiers

Krzysztof Rajda, Lukasz Augustyniak, Piotr Gramacki, Marcin Gruza, Szymon Woźniak, Tomasz Kajdanowicz


Abstract
Models are increasing in size and complexity in the hunt for SOTA. But what if those 2%increase in performance does not make a difference in a production use case? Maybe benefits from a smaller, faster model outweigh those slight performance gains. Also, equally good performance across languages in multilingual tasks is more important than SOTA results on a single one. We present the biggest, unified, multilingual collection of sentiment analysis datasets. We use these to assess 11 models and 80 high-quality sentiment datasets (out of 342 raw datasets collected) in 27 languages and included results on the internally annotated datasets. We deeply evaluate multiple setups, including fine-tuning transformer-based models for measuring performance. We compare results in numerous dimensions addressing the imbalance in both languages coverage and dataset sizes. Finally, we present some best practices for working with such a massive collection of datasets and models for a multi-lingual perspective.
Anthology ID:
2022.wassa-1.13
Volume:
Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
WASSA
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
125–140
Language:
URL:
https://aclanthology.org/2022.wassa-1.13
DOI:
10.18653/v1/2022.wassa-1.13
Bibkey:
Cite (ACL):
Krzysztof Rajda, Lukasz Augustyniak, Piotr Gramacki, Marcin Gruza, Szymon Woźniak, and Tomasz Kajdanowicz. 2022. Assessment of Massively Multilingual Sentiment Classifiers. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 125–140, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Assessment of Massively Multilingual Sentiment Classifiers (Rajda et al., WASSA 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.wassa-1.13.pdf
Video:
 https://preview.aclanthology.org/ingestion-script-update/2022.wassa-1.13.mp4