Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

Michael A. Hedderich, David Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, Dietrich Klakow

Abstract
Multilingual transformer models like mBERT and XLM-RoBERTa have obtained great improvements for many NLP tasks on a variety of languages. However, recent works also showed that results from high-resource languages could not be easily transferred to realistic, low-resource scenarios. In this work, we study trends in performance for different amounts of available resources for the three African languages Hausa, isiXhosa and on both NER and topic classification. We show that in combination with transfer learning or distant supervision, these models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. However, we also find settings where this does not hold. Our discussions and additional experiments on assumptions such as time and hardware restrictions highlight challenges and opportunities in low-resource learning.
Anthology ID:
2020.emnlp-main.204
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2580–2591
Language:
URL:
https://aclanthology.org/2020.emnlp-main.204
DOI:
10.18653/v1/2020.emnlp-main.204
Bibkey:
Cite (ACL):
Michael A. Hedderich, David Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, and Dietrich Klakow. 2020. Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2580–2591, Online. Association for Computational Linguistics.
Cite (Informal):
Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages (Hedderich et al., EMNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/starsem-semeval-split/2020.emnlp-main.204.pdf
Optional supplementary material:
 2020.emnlp-main.204.OptionalSupplementaryMaterial.pdf
Video:
 https://slideslive.com/38938875
Code
 uds-lsv/transfer-distant-transformer-african
PDF Search Code Optional supplementary material Video