A Neural Lemmatizer for Bengali

Abhisek Chakrabarty, Akshay Chaturvedi, Utpal Garain

Abstract
We propose a novel neural lemmatization model which is language independent and supervised in nature. To handle the words in a neural framework, word embedding technique is used to represent words as vectors. The proposed lemmatizer makes use of contextual information of the surface word to be lemmatized. Given a word along with its contextual neighbours as input, the model is designed to produce the lemma of the concerned word as output. We introduce a new network architecture that permits only dimension specific connections between the input and the output layer of the model. For the present work, Bengali is taken as the reference language. Two datasets are prepared for training and testing purpose consisting of 19,159 and 2,126 instances respectively. As Bengali is a resource scarce language, these datasets would be beneficial for the respective research community. Evaluation method shows that the neural lemmatizer achieves 69.57% accuracy on the test dataset and outperforms the simple cosine similarity based baseline strategy by a margin of 1.37%.
Anthology ID:
L16-1406
Volume:
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
Month:
May
Year:
2016
Address:
Portorož, Slovenia
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
2558–2561
Language:
URL:
https://aclanthology.org/L16-1406
DOI:
Bibkey:
Cite (ACL):
Abhisek Chakrabarty, Akshay Chaturvedi, and Utpal Garain. 2016. A Neural Lemmatizer for Bengali. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 2558–2561, Portorož, Slovenia. European Language Resources Association (ELRA).
Cite (Informal):
A Neural Lemmatizer for Bengali (Chakrabarty et al., LREC 2016)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-1/L16-1406.pdf
PDF Search