Corpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text

Siva Subrahamanyam Varma Kusampudi, Anudeep Chaluvadi, Radhika Mamidi

Abstract
Code-Mixing (CM) is a common phenomenon in multilingual societies. CM plays a significant role in technology and medical fields where terminologies in the native language are not available or known. Language Identification (LID) of the CM data will help solve NLP tasks such as Spell Checking, Named Entity Recognition, Part-Of-Speech tagging, and Semantic Parsing. In the current era of machine learning, a common problem to the above-mentioned tasks is the availability of Learning data to train models. In this paper, we introduce two Telugu-English CM manually annotated datasets (Twitter dataset and Blog dataset). The Twitter dataset contains more romanization variability and misspelled words than the blog dataset. We compare across various classification models and perform extensive bench-marking using both Classical and Deep Learning Models for LID compared to existing models. We propose two architectures for language classification (Telugu and English) in CM data: (1) Word Level Classification (2) Sentence Level word-by-word Classification and compare these approaches presenting two strong baselines for LID on these datasets.
Anthology ID:
2021.ranlp-1.85
Volume:
Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
Month:
September
Year:
2021
Address:
Held Online
Editors:
Ruslan Mitkov, Galia Angelova
Venue:
RANLP
SIG:
Publisher:
INCOMA Ltd.
Note:
Pages:
744–752
Language:
URL:
https://aclanthology.org/2021.ranlp-1.85
DOI:
Bibkey:
Cite (ACL):
Siva Subrahamanyam Varma Kusampudi, Anudeep Chaluvadi, and Radhika Mamidi. 2021. Corpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 744–752, Held Online. INCOMA Ltd..
Cite (Informal):
Corpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text (Kusampudi et al., RANLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp22-frontmatter/2021.ranlp-1.85.pdf
PDF Search