@inproceedings{almatham-etal-2025-balsam,
    title = "{BALSAM}: A Platform for Benchmarking {A}rabic Large Language Models",
    author = "Almatham, Rawan Nasser  and
      Darwish, Kareem Mohamed  and
      Al-Rasheed, Raghad  and
      Alshammari, Waad Thuwaini  and
      Alhoshan, Muneera  and
      Almazrua, Amal  and
      Al Wazrah, Asma  and
      Alheraki, Mais  and
      Alam, Firoj  and
      Nakov, Preslav  and
      Alzahrani, Norah A.  and
      Albilali, Eman  and
      Habash, Nizar  and
      El-Sheikh, Abdelrahman Mustafa  and
      Elmallah, Muhammad  and
      Mubarak, Hamdy  and
      Alyafeai, Zaid  and
      Anwar, Mohamed  and
      Li, Haonan  and
      Abdelali, Ahmed  and
      Altwairesh, Nora  and
      Hasanain, Maram  and
      Al-Thubaity, Abdulmohsen  and
      Shehata, Shady  and
      Alhafni, Bashar  and
      Hamed, Injy  and
      Inoue, Go  and
      Elmadani, Khalid N.  and
      Obeid, Ossama  and
      Haouari, Fatima  and
      Elsayed, Tamer  and
      Alghamdi, Emad A.  and
      Almubarak, Khalid  and
      Alshahrani, Saied  and
      Aljareh, Ola  and
      Alajlan, Safa  and
      Alshaqarawi, Areej  and
      Alshihri, Maryam  and
      Alghurabi, Sultana  and
      Alzeghayer, Atikah  and
      Altamimi, Afrah  and
      Alfaifi, Abdullah  and
      Alosaimy, Abdulrahman M",
    editor = "Darwish, Kareem  and
      Ali, Ahmed  and
      Abu Farha, Ibrahim  and
      Touileb, Samia  and
      Zitouni, Imed  and
      Abdelali, Ahmed  and
      Al-Ghamdi, Sharefah  and
      Alkhereyf, Sakhar  and
      Zaghouani, Wajdi  and
      Khalifa, Salam  and
      AlKhamissi, Badr  and
      Almatham, Rawan  and
      Hamed, Injy  and
      Alyafeai, Zaid  and
      Alowisheq, Areeb  and
      Inoue, Go  and
      Mrini, Khalil  and
      Alshammari, Waad",
    booktitle = "Proceedings of The Third Arabic Natural Language Processing Conference",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.arabicnlp-main.21/",
    pages = "258--277",
    ISBN = "979-8-89176-352-4",
    abstract = "The impressive advancement of Large Language Models (LLMs) in English has not been matched across all languages. In particular, LLM performance in Arabic lags behind, due to data scarcity, linguistic diversity of Arabic and its dialects, morphological complexity, etc. Progress is further hindered by the quality of Arabic benchmarks, which typically rely on static, publicly available data, lack comprehensive task coverage, or do not provide dedicated platforms with blind test sets. This makes it challenging to measure actual progress and to mitigate data contamination. Here, we aim to bridge these gaps. In particular, we introduce BALSAM, a comprehensive, community-driven benchmark aimed at advancing Arabic LLM development and evaluation. It includes 78 NLP tasks from 14 broad categories, with 52K examples divided into 37K test and 15K development, and a centralized, transparent platform for blind evaluation. We envision BALSAM as a unifying platform that sets standards and promotes collaborative research to advance Arabic LLM capabilities."
}