Automatic Detection of Arabicized Berber and Arabic Varieties
Wafia Adouane, Nasredine Semmar, Richard Johansson, Victoria Bobicev
Abstract
Automatic Language Identification (ALI) is the detection of the natural language of an input text by a machine. It is the first necessary step to do any language-dependent natural language processing task. Various methods have been successfully applied to a wide range of languages, and the state-of-the-art automatic language identifiers are mainly based on character n-gram models trained on huge corpora. However, there are many languages which are not yet automatically processed, for instance minority and informal languages. Many of these languages are only spoken and do not exist in a written format. Social media platforms and new technologies have facilitated the emergence of written format for these spoken languages based on pronunciation. The latter are not well represented on the Web, commonly referred to as under-resourced languages, and the current available ALI tools fail to properly recognize them. In this paper, we revisit the problem of ALI with the focus on Arabicized Berber and dialectal Arabic short texts. We introduce new resources and evaluate the existing methods. The results show that machine learning models combined with lexicons are well suited for detecting Arabicized Berber and different Arabic varieties and distinguishing between them, giving a macro-average F-score of 92.94%.- Anthology ID:
- W16-4809
- Volume:
- Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)
- Month:
- December
- Year:
- 2016
- Address:
- Osaka, Japan
- Editors:
- Preslav Nakov, Marcos Zampieri, Liling Tan, Nikola Ljubešić, Jörg Tiedemann, Shervin Malmasi
- Venue:
- VarDial
- SIG:
- Publisher:
- The COLING 2016 Organizing Committee
- Note:
- Pages:
- 63–72
- Language:
- URL:
- https://aclanthology.org/W16-4809
- DOI:
- Cite (ACL):
- Wafia Adouane, Nasredine Semmar, Richard Johansson, and Victoria Bobicev. 2016. Automatic Detection of Arabicized Berber and Arabic Varieties. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 63–72, Osaka, Japan. The COLING 2016 Organizing Committee.
- Cite (Informal):
- Automatic Detection of Arabicized Berber and Arabic Varieties (Adouane et al., VarDial 2016)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/W16-4809.pdf