Exploiting Dialect Identification in Automatic Dialectal Text Normalization
Bashar Alhafni, Sarah Al-Towaity, Ziyad Fawzy, Fatema Nassar, Fadhl Eryani, Houda Bouamor, Nizar Habash
Abstract
Dialectal Arabic is the primary spoken language used by native Arabic speakers in daily communication. The rise of social media platforms has notably expanded its use as a written language. However, Arabic dialects do not have standard orthographies. This, combined with the inherent noise in user-generated content on social media, presents a major challenge to NLP applications dealing with Dialectal Arabic. In this paper, we explore and report on the task of CODAfication, which aims to normalize Dialectal Arabic into the Conventional Orthography for Dialectal Arabic (CODA). We work with a unique parallel corpus of multiple Arabic dialects focusing on five major city dialects. We benchmark newly developed pretrained sequence-to-sequence models on the task of CODAfication. We further show that using dialect identification information improves the performance across all dialects. We make our code, data, andpretrained models publicly available.- Anthology ID:
- 2024.arabicnlp-1.4
- Volume:
- Proceedings of The Second Arabic Natural Language Processing Conference
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Nizar Habash, Houda Bouamor, Ramy Eskander, Nadi Tomeh, Ibrahim Abu Farha, Ahmed Abdelali, Samia Touileb, Injy Hamed, Yaser Onaizan, Bashar Alhafni, Wissam Antoun, Salam Khalifa, Hatem Haddad, Imed Zitouni, Badr AlKhamissi, Rawan Almatham, Khalil Mrini
- Venues:
- ArabicNLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 42–54
- Language:
- URL:
- https://aclanthology.org/2024.arabicnlp-1.4
- DOI:
- 10.18653/v1/2024.arabicnlp-1.4
- Cite (ACL):
- Bashar Alhafni, Sarah Al-Towaity, Ziyad Fawzy, Fatema Nassar, Fadhl Eryani, Houda Bouamor, and Nizar Habash. 2024. Exploiting Dialect Identification in Automatic Dialectal Text Normalization. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 42–54, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Exploiting Dialect Identification in Automatic Dialectal Text Normalization (Alhafni et al., ArabicNLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2024.arabicnlp-1.4.pdf