Abstract
Linguistic variability poses a challenge to many modern ASR systems, particularly Dialectical Arabic (DA) ASR systems dealing with low-resource dialects and resulting morphological and orthographic variations in text and speech. Traditional evaluation metrics such as the word error rate (WER) inadequately capture these complexities, leading to an incomplete assessment of DA ASR performance. We propose AraDiaWER, an ASR evaluation metric for Dialectical Arabic (DA) speech recognition systems, focused on the Egyptian dialect. AraDiaWER uses language model embeddings for the syntactic and semantic aspects of ASR errors to identify their root cause, not captured by traditional WER. MiniLM generates the semantic score, capturing contextual differences between reference and predicted transcripts. CAMeLBERT-Mix assigns morphological and lexical tags using a fuzzy matching algorithm to calculate the syntactic score. Our experiments validate the effectiveness of AraDiaWER. By incorporating language model embeddings, AraDiaWER enables a more interpretable evaluation, allowing us to improve DA ASR systems. We position the proposed metric as a complementary tool to WER, capturing syntactic and semantic features not represented by WER. Additionally, we use UMAP analysis to observe the quality of ASR embeddings in the proposed evaluation framework.- Anthology ID:
- 2023.fieldmatters-1.8
- Volume:
- Proceedings of the Second Workshop on NLP Applications to Field Linguistics
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Oleg Serikov, Ekaterina Voloshina, Anna Postnikova, Elena Klyachko, Ekaterina Vylomova, Tatiana Shavrina, Eric Le Ferrand, Valentin Malykh, Francis Tyers, Timofey Arkhangelskiy, Vladislav Mikhailov
- Venue:
- FieldMatters
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 64–73
- Language:
- URL:
- https://aclanthology.org/2023.fieldmatters-1.8
- DOI:
- 10.18653/v1/2023.fieldmatters-1.8
- Cite (ACL):
- Abdulwahab Sahyoun and Shady Shehata. 2023. AraDiaWER: An Explainable Metric For Dialectical Arabic ASR. In Proceedings of the Second Workshop on NLP Applications to Field Linguistics, pages 64–73, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- AraDiaWER: An Explainable Metric For Dialectical Arabic ASR (Sahyoun & Shehata, FieldMatters 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.fieldmatters-1.8.pdf