Abstract
Code-mixed text infused with low resource language has always been a challenge for natural language understanding models. A significant problem while understanding such texts is the correlation between the syntactic and semantic arrangement of words. The phonemes of each character in a word dictates the spelling representation of a term in low resource language. However, there is no universal protocol or alphabet mapping for code-mixing. In this paper, we highlight the impact of spelling variations in code-mixed data for training natural language understanding models. We emphasize the impact of using phonetics to neutralize this variation in spelling across different usage of a word with the same semantics. The proposed approach is a computationally inexpensive technique and improves the performances of state-of-the-art models for three dialog system tasks viz. intent classification, slot-filling, and response generation. We propose a data pipeline for normalizing spelling variations irrespective of language.- Anthology ID:
- 2022.icon-main.33
- Volume:
- Proceedings of the 19th International Conference on Natural Language Processing (ICON)
- Month:
- December
- Year:
- 2022
- Address:
- New Delhi, India
- Editors:
- Md. Shad Akhtar, Tanmoy Chakraborty
- Venue:
- ICON
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 269–279
- Language:
- URL:
- https://aclanthology.org/2022.icon-main.33
- DOI:
- Cite (ACL):
- Krishna Yadav, Md Akhtar, and Tanmoy Chakraborty. 2022. Normalization of Spelling Variations in Code-Mixed Data. In Proceedings of the 19th International Conference on Natural Language Processing (ICON), pages 269–279, New Delhi, India. Association for Computational Linguistics.
- Cite (Informal):
- Normalization of Spelling Variations in Code-Mixed Data (Yadav et al., ICON 2022)
- PDF:
- https://preview.aclanthology.org/revert-3132-ingestion-checklist/2022.icon-main.33.pdf