@inproceedings{moosavi-monazzah-etal-2025-percul,
    title = "{P}er{C}ul: A Story-Driven Cultural Evaluation of {LLM}s in {P}ersian",
    author = "Moosavi Monazzah, Erfan  and
      Rahimzadeh, Vahid  and
      Yaghoobzadeh, Yadollah  and
      Shakery, Azadeh  and
      Pilehvar, Mohammad Taher",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.naacl-long.631/",
    pages = "12670--12687",
    ISBN = "979-8-89176-189-6",
    abstract = "Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios.Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut. We evaluate several state-of-the-art multilingual and Persian-specific LLMs, establishing a foundation for future research in cross-cultural NLP evaluation. Our experiments demonstrate a 11.3{\%} gap between best closed source model and layperson baseline while the gap increases to 21.3{\%} by using the best open-weight model. You can access the dataset from here:https://huggingface.co/datasets/teias-ai/percul"
}