Named Entity Recognition for Hindi-English Code-Mixed Social Media Text

Vinay Singh, Deepanshu Vijay, Syed Sarfaraz Akhtar, Manish Shrivastava

Abstract
Named Entity Recognition (NER) is a major task in the field of Natural Language Processing (NLP), and also is a sub-task of Information Extraction. The challenge of NER for tweets lie in the insufficient information available in a tweet. There has been a significant amount of work done related to entity extraction, but only for resource rich languages and domains such as newswire. Entity extraction is, in general, a challenging task for such an informal text, and code-mixed text further complicates the process with it’s unstructured and incomplete information. We propose experiments with different machine learning classification algorithms with word, character and lexical features. The algorithms we experimented with are Decision tree, Long Short-Term Memory (LSTM), and Conditional Random Field (CRF). In this paper, we present a corpus for NER in Hindi-English Code-Mixed along with extensive experiments on our machine learning models which achieved the best f1-score of 0.95 with both CRF and LSTM.
Anthology ID:
W18-2405
Volume:
Proceedings of the Seventh Named Entities Workshop
Month:
July
Year:
2018
Address:
Melbourne, Australia
Venue:
NEWS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
27–35
Language:
URL:
https://aclanthology.org/W18-2405
DOI:
10.18653/v1/W18-2405
Bibkey:
Cite (ACL):
Vinay Singh, Deepanshu Vijay, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. Named Entity Recognition for Hindi-English Code-Mixed Social Media Text. In Proceedings of the Seventh Named Entities Workshop, pages 27–35, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Named Entity Recognition for Hindi-English Code-Mixed Social Media Text (Singh et al., NEWS 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/remove-xml-comments/W18-2405.pdf
Code
 SilentFlame/Named-Entity-Recognition
PDF Search Code