@inproceedings{ghaboura-etal-2025-camel,
    title = "{CAMEL}-Bench: A Comprehensive {A}rabic {LMM} Benchmark",
    author = "Ghaboura, Sara  and
      Heakl, Ahmed  and
      Thawakar, Omkar  and
      Alharthi, Ali Husain Salem Abdulla  and
      Riahi, Ines  and
      Radman, Abduljalil  and
      Laaksonen, Jorma  and
      Khan, Fahad Shahbaz  and
      Khan, Salman  and
      Anwer, Rao Muhammad",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.105/",
    pages = "1970--1980",
    ISBN = "979-8-89176-195-7",
    abstract = "Recent years have witnessed a significant interest in developing large multi-modal models (LMMs) capable of performing various visual reasoning and understanding tasks. This has led to the introduction of multiple LMM benchmarks to evaluate LMMs on different tasks. However, most existing LMM evaluation benchmarks are predominantly English-centric. In this work, we develop a comprehensive LMM evaluation benchmark for the Arabic language to represent a large population of over 400 million speakers. The proposed benchmark, named CAMEL-Bench, comprises eight diverse domains and 38 sub-domains including, multi-image understanding, complex visual perception, handwritten document understanding, video understanding, medical imaging, plant diseases, and remote sensing-based land use understanding to evaluate broad scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions that are filtered from a larger pool of samples, where the quality is manually verified by native speakers to ensure reliable model assessment. We conduct evaluations of both closed-source, including GPT-4 series, and open-source LMMs. Our analysis reveals the need for substantial improvement, especially among the bestopen-source models, with even the closed-source GPT-4o achieving an overall score of 62{\%}. Our benchmark will be publicly released."
}