Abstract
We present ArTrivia, a new Arabic question-answering dataset consisting of more than 10,000 question-answer pairs along with relevant passages, covering a wide range of 18 diverse topics in Arabic. We created our dataset using a newly proposed pipeline that leverages diverse structured data sources from Arabic Wikipedia. Moreover, we conducted a comprehensive statistical analysis of ArTrivia and assessed the performance of each component in our pipeline. Additionally, we compared the performance of ArTrivia against the existing TyDi QA dataset using various experimental setups. Our analysis highlights the significance of often overlooked aspects in dataset creation, such as answer normalization, in enhancing the quality of QA datasets. Our evaluation also shows that ArTrivia presents more challenging and out-of-distribution questions to TyDi, raising questions about the feasibility of using ArTrivia as a complementary dataset to TyDi.- Anthology ID:
- 2023.arabicnlp-1.17
- Volume:
- Proceedings of ArabicNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore (Hybrid)
- Editors:
- Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, Rawan Almatham
- Venues:
- ArabicNLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 191–207
- Language:
- URL:
- https://aclanthology.org/2023.arabicnlp-1.17
- DOI:
- 10.18653/v1/2023.arabicnlp-1.17
- Cite (ACL):
- Sultan Alrowili and K Vijay-Shanker. 2023. ArTrivia: Harvesting Arabic Wikipedia to Build A New Arabic Question Answering Dataset. In Proceedings of ArabicNLP 2023, pages 191–207, Singapore (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- ArTrivia: Harvesting Arabic Wikipedia to Build A New Arabic Question Answering Dataset (Alrowili & Vijay-Shanker, ArabicNLP-WS 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/2023.arabicnlp-1.17.pdf