Abstract
Code-mixing is a phenomenon of mixing words and phrases from two or more languages in a single utterance of speech and text. Due to the high linguistic diversity, code-mixing presents several challenges in evaluating standard natural language generation (NLG) tasks. Various widely popular metrics perform poorly with the code-mixed NLG tasks. To address this challenge, we present a metric in- dependent evaluation pipeline MIPE that significantly improves the correlation between evaluation metrics and human judgments on the generated code-mixed text. As a use case, we demonstrate the performance of MIPE on the machine-generated Hinglish (code-mixing of Hindi and English languages) sentences from the HinGE corpus. We can extend the proposed evaluation strategy to other code-mixed language pairs, NLG tasks, and evaluation metrics with minimal to no effort.- Anthology ID:
- 2021.eval4nlp-1.13
- Volume:
- Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems
- Month:
- November
- Year:
- 2021
- Address:
- Punta Cana, Dominican Republic
- Editors:
- Yang Gao, Steffen Eger, Wei Zhao, Piyawat Lertvittayakumjorn, Marina Fomicheva
- Venue:
- Eval4NLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 123–132
- Language:
- URL:
- https://aclanthology.org/2021.eval4nlp-1.13
- DOI:
- 10.18653/v1/2021.eval4nlp-1.13
- Cite (ACL):
- Ayush Garg, Sammed Kagi, Vivek Srivastava, and Mayank Singh. 2021. MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 123–132, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation (Garg et al., Eval4NLP 2021)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2021.eval4nlp-1.13.pdf