@inproceedings{althnian-etal-2025-araeval,
    title = "{A}ra{E}val: An {A}rabic Multi-Task Evaluation Suite for Large Language Models",
    author = "Althnian, Alhanoof  and
      Alzahrani, Norah A.  and
      Alsubaie, Shaykhah Z.  and
      Albilali, Eman  and
      Abdelali, Ahmed  and
      Alotaibi, Nouf M.  and
      Bari, M Saiful  and
      Alnumay, Yazeed  and
      Alothaimen, Abdulhamed  and
      Saif, Maryam  and
      Alzaidi, Shahad D.  and
      Mirza, Faisal Abdulrahman  and
      Almushayqih, Yousef  and
      Al Saleem, Mohammed  and
      Alabduljabbar, Ghadah  and
      Al-Thubaity, Abdulmohsen  and
      Alowisheq, Areeb  and
      Al-Twairesh, Nora",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1679/",
    pages = "33025--33049",
    ISBN = "979-8-89176-332-6",
    abstract = "The rapid advancements of Large Language models (LLMs) necessitate robust benchmarks. In this paper, we present AraEval, a pioneering and comprehensive evaluation suite specifically developed to assess the advanced knowledge, reasoning, truthfulness, and instruction- following capabilities of foundation models in the Arabic context. AraEval includes a diverse set of evaluation tasks that test various dimensions of knowledge and reasoning, with a total of 24,378 samples. These tasks cover areas such as linguistic understanding, factual recall, logical inference, commonsense reasoning, mathematical problem-solving, and domain-specific expertise, ensuring that the evaluation goes beyond basic language comprehension. It covers multiple domains of knowledge, such as science, history, religion, and literature, ensuring that the LLMs are tested on a broad spectrum of topics relevant to Arabic-speaking contexts. AraEval is designed to facilitate comparisons across different foundation models, enabling LLM developers and users to benchmark perfor- mance effectively. In addition, it provides diagnostic insights to identify specific areas where models excel or struggle, guiding further development. AraEval datasets can be found at https://huggingface.co/collections/humain-ai/araeval-datasets-687760e04b12a7afb429a4a0."
}