Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing

Ganesh Jawahar, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S.


Abstract
We describe models focused at the understudied problem of translating between monolingual and code-mixed language pairs. More specifically, we offer a wide range of models that convert monolingual English text into Hinglish (code-mixed Hindi and English). Given the recent success of pretrained language models, we also test the utility of two recent Transformer-based encoder-decoder models (i.e., mT5 and mBART) on the task finding both to work well. Given the paucity of training data for code-mixing, we also propose a dependency-free method for generating code-mixed texts from bilingual distributed representations that we exploit for improving language model performance. In particular, armed with this additional data, we adopt a curriculum learning approach where we first finetune the language models on synthetic data then on gold code-mixed data. We find that, although simple, our synthetic code-mixing method is competitive with (and in some cases is even superior to) several standard methods (backtranslation, method based on equivalence constraint theory) under a diverse set of conditions. Our work shows that the mT5 model, finetuned following the curriculum learning procedure, achieves best translation performance (12.67 BLEU). Our models place first in the overall ranking of the English-Hinglish official shared task.
Anthology ID:
2021.calcs-1.6
Volume:
Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching
Month:
June
Year:
2021
Address:
Online
Venue:
CALCS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
36–46
Language:
URL:
https://aclanthology.org/2021.calcs-1.6
DOI:
10.18653/v1/2021.calcs-1.6
Bibkey:
Cite (ACL):
Ganesh Jawahar, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S.. 2021. Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing. In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 36–46, Online. Association for Computational Linguistics.
Cite (Informal):
Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing (Jawahar et al., CALCS 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2021.calcs-1.6.pdf
Data
PHINC