Detection of Similar Languages and Dialects Using Deep Supervised Autoencoder
Shantipriya Parida, Esau Villatoro-Tello, Sajit Kumar, Maël Fabien, Petr Motlicek
Abstract
Language detection is considered a difficult task especially for similar languages, varieties, and dialects. With the growing number of online content in different languages, the need for reliable and robust language detection tools also increased. In this work, we use supervised autoencoders with a bayesian optimizer for language detection and highlights its efficiency in detecting similar languages with dialect variance in comparison to other state-of-the-art techniques. We evaluated our approach on multiple datasets (Ling10, Discriminating between Similar Language (DSL), and Indo-Aryan Language Identification (ILI)). Obtained results demonstrate that SAE are higly effective in detecting languages, up to a 100% accuracy in the Ling10. Similarly, we obtain a competitive performance in identifying similar languages, and dialects, 92% and 85% for DSL ans ILI datasets respectively.- Anthology ID:
- 2020.icon-main.49
- Volume:
- Proceedings of the 17th International Conference on Natural Language Processing (ICON)
- Month:
- December
- Year:
- 2020
- Address:
- Indian Institute of Technology Patna, Patna, India
- Editors:
- Pushpak Bhattacharyya, Dipti Misra Sharma, Rajeev Sangal
- Venue:
- ICON
- SIG:
- Publisher:
- NLP Association of India (NLPAI)
- Note:
- Pages:
- 362–367
- Language:
- URL:
- https://aclanthology.org/2020.icon-main.49
- DOI:
- Cite (ACL):
- Shantipriya Parida, Esau Villatoro-Tello, Sajit Kumar, Maël Fabien, and Petr Motlicek. 2020. Detection of Similar Languages and Dialects Using Deep Supervised Autoencoder. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pages 362–367, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI).
- Cite (Informal):
- Detection of Similar Languages and Dialects Using Deep Supervised Autoencoder (Parida et al., ICON 2020)
- PDF:
- https://preview.aclanthology.org/bionlp-24-ingestion/2020.icon-main.49.pdf