@inproceedings{elsetohy-etal-2026-macaron,
    title = "Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling",
    author = "Elsetohy, Alaa  and
      Hadhoud, Sama  and
      Wibowo, Haryo Akbarianto  and
      Whitehouse, Chenxi  and
      Winata, Genta Indra  and
      Koto, Fajri  and
      Aji, Alham Fikri",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.2211/",
    pages = "47885--47906",
    ISBN = "979-8-89176-390-6",
    abstract = "Multilingual benchmarks rarely test reasoning over culturally grounded premises: translated datasets keep English-centric scenarios, while culture-first datasets often lack control over the reasoning required. We propose Macaron, a template-first benchmark that factorizes reasoning type and cultural aspect across question languages. Using 100 language-agnostic templates that cover 7 reasoning types, 22 cultural aspects, native annotators create scenario-aligned English and local-language multiple-choice questions, and systematically derived True/False questions. Macaron contains 11,862 instances spanning 20 countries/cultural contexts, 10 scripts, and 20 languages and dialects (including low-resource ones like Amharic, Yoruba, Zulu, Kyrgyz, and some Arabic dialects). In zero-shot evaluation of 21 multilingual LLMs, reasoning-mode models achieve the strongest performance (80.8{\%} overall) and near-parity between English and local languages ({\ensuremath{\Delta}}MC = {\ensuremath{-}}1.3{\%}), while open-weight models degrade substantially in local languages ({\ensuremath{\Delta}}MC = {\ensuremath{-}}6.8{\%}) and often approach chance on T/F tasks. Culture-grounded mathematical and counting templates are consistently the hardest. The data can be accessed here https://huggingface.co/datasets/AlaaAhmed2444/Macaron."
}