@inproceedings{garg-etal-2021-mipe,
    title = "{MIPE}: A Metric Independent Pipeline for Effective Code-Mixed {NLG} Evaluation",
    author = "Garg, Ayush  and
      Kagi, Sammed  and
      Srivastava, Vivek  and
      Singh, Mayank",
    editor = "Gao, Yang  and
      Eger, Steffen  and
      Zhao, Wei  and
      Lertvittayakumjorn, Piyawat  and
      Fomicheva, Marina",
    booktitle = "Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2021.eval4nlp-1.13/",
    doi = "10.18653/v1/2021.eval4nlp-1.13",
    pages = "123--132",
    abstract = "Code-mixing is a phenomenon of mixing words and phrases from two or more languages in a single utterance of speech and text. Due to the high linguistic diversity, code-mixing presents several challenges in evaluating standard natural language generation (NLG) tasks. Various widely popular metrics perform poorly with the code-mixed NLG tasks. To address this challenge, we present a metric in- dependent evaluation pipeline MIPE that significantly improves the correlation between evaluation metrics and human judgments on the generated code-mixed text. As a use case, we demonstrate the performance of MIPE on the machine-generated Hinglish (code-mixing of Hindi and English languages) sentences from the HinGE corpus. We can extend the proposed evaluation strategy to other code-mixed language pairs, NLG tasks, and evaluation metrics with minimal to no effort."
}