Corpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text
Siva Subrahamanyam Varma Kusampudi, Anudeep Chaluvadi, Radhika Mamidi
Abstract
Code-Mixing (CM) is a common phenomenon in multilingual societies. CM plays a significant role in technology and medical fields where terminologies in the native language are not available or known. Language Identification (LID) of the CM data will help solve NLP tasks such as Spell Checking, Named Entity Recognition, Part-Of-Speech tagging, and Semantic Parsing. In the current era of machine learning, a common problem to the above-mentioned tasks is the availability of Learning data to train models. In this paper, we introduce two Telugu-English CM manually annotated datasets (Twitter dataset and Blog dataset). The Twitter dataset contains more romanization variability and misspelled words than the blog dataset. We compare across various classification models and perform extensive bench-marking using both Classical and Deep Learning Models for LID compared to existing models. We propose two architectures for language classification (Telugu and English) in CM data: (1) Word Level Classification (2) Sentence Level word-by-word Classification and compare these approaches presenting two strong baselines for LID on these datasets.- Anthology ID:
- 2021.ranlp-1.85
- Volume:
- Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
- Month:
- September
- Year:
- 2021
- Address:
- Held Online
- Editors:
- Ruslan Mitkov, Galia Angelova
- Venue:
- RANLP
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 744–752
- Language:
- URL:
- https://aclanthology.org/2021.ranlp-1.85
- DOI:
- Cite (ACL):
- Siva Subrahamanyam Varma Kusampudi, Anudeep Chaluvadi, and Radhika Mamidi. 2021. Corpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 744–752, Held Online. INCOMA Ltd..
- Cite (Informal):
- Corpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text (Kusampudi et al., RANLP 2021)
- PDF:
- https://preview.aclanthology.org/bionlp-24-ingestion/2021.ranlp-1.85.pdf