PreCogIIITH at HinglishEval : Leveraging Code-Mixing Metrics & Language Model Embeddings To Estimate Code-Mix Quality
Prashant Kodali, Tanmay Sachan, Akshay Goindani, Anmol Goel, Naman Ahuja, Manish Shrivastava, Ponnurangam Kumaraguru
Abstract
Code-Mixing is a phenomenon of mixing two or more languages in a speech event and is prevalent in multilingual societies. Given the low-resource nature of Code-Mixing, machine generation of code-mixed text is a prevalent approach for data augmentation. However, evaluating the quality of such machine gen- erated code-mixed text is an open problem. In our submission to HinglishEval, a shared- task collocated with INLG2022, we attempt to build models factors that impact the quality of synthetically generated code-mix text by pre- dicting ratings for code-mix quality. Hingli- shEval Shared Task consists of two sub-tasks - a) Quality rating prediction); b) Disagree- ment prediction. We leverage popular code- mixed metrics and embeddings of multilin- gual large language models (MLLMs) as fea- tures, and train task specific MLP regression models. Our approach could not beat the baseline results. However, for Subtask-A our team ranked a close second on F-1 and Co- hen’s Kappa Score measures and first for Mean Squared Error measure. For Subtask-B our ap- proach ranked third for F1 score, and first for Mean Squared Error measure. Code of our submission can be accessed here.- Anthology ID:
- 2022.inlg-genchal.4
- Volume:
- Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges
- Month:
- July
- Year:
- 2022
- Address:
- Waterville, Maine, USA and virtual meeting
- Editors:
- Samira Shaikh, Thiago Ferreira, Amanda Stent
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 26–30
- Language:
- URL:
- https://aclanthology.org/2022.inlg-genchal.4
- DOI:
- Cite (ACL):
- Prashant Kodali, Tanmay Sachan, Akshay Goindani, Anmol Goel, Naman Ahuja, Manish Shrivastava, and Ponnurangam Kumaraguru. 2022. PreCogIIITH at HinglishEval : Leveraging Code-Mixing Metrics & Language Model Embeddings To Estimate Code-Mix Quality. In Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges, pages 26–30, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
- Cite (Informal):
- PreCogIIITH at HinglishEval : Leveraging Code-Mixing Metrics & Language Model Embeddings To Estimate Code-Mix Quality (Kodali et al., INLG 2022)
- PDF:
- https://preview.aclanthology.org/improve-issue-templates/2022.inlg-genchal.4.pdf