@inproceedings{al-kautsar-etal-2026-cultural,
    title = "Cultural Benchmarking of {LLM}s in Standard and Dialectal {A}rabic Dialogues",
    author = "Al Kautsar, Muhammad Dehan  and
      Almheiri, Saeed  and
      Ahsan, Momina  and
      Elbouardi, Bilal  and
      Samih, Younes  and
      Ahmad, Sarfraz  and
      Keleg, Amr  and
      El Herraoui, Omar  and
      Elzeky, Kareem  and
      Freihat, Abed Alhakim  and
      Anwar, Mohamed  and
      Xie, Zhuohan  and
      Liang, Junhong  and
      Al Nasar, Mohammad Rustom  and
      Nakov, Preslav  and
      Koto, Fajri",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.963/",
    pages = "21016--21039",
    ISBN = "979-8-89176-390-6",
    abstract = "There is a significant gap in evaluating cultural reasoning in LLMs using conversational datasets that capture culturally rich and dialectal contexts. Most Arabic benchmarks focus on short text snippets in Modern Standard Arabic (MSA), overlooking the cultural nuances that naturally arise in dialogues. To address this gap, we introduce ArabCulture-Dialogue, a culturally grounded conversational dataset covering 13 Arabic-speaking countries, in both MSA and each country{'}s respective dialect, spanning 12 daily-life topics and 54 fine-grained subtopics. We utilize the dataset to form three benchmarking tasks: (i) multiple-choice cultural reasoning, (ii) machine translation between MSA and dialects, and (iii) dialect-steering generation. Our experiments indicate that the performance gap between MSA and Arabic dialects still exists, whereby the models perform worse on all three tasks in the dialectal setup, compared to the MSA one."
}