@inproceedings{hosseinbeigi-etal-2025-advancing,
    title = "Advancing {P}ersian {LLM} Evaluation",
    author = "Hosseinbeigi, Sara Bourbour  and
      Rohani, Behnam  and
      Masoudi, Mostafa  and
      Shamsfard, Mehrnoush  and
      Saaberi, Zahra  and
      Manesh, Mostafa Karimi  and
      Abbasi, Mohammad Amin",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.147/",
    pages = "2711--2727",
    ISBN = "979-8-89176-195-7",
    abstract = "Evaluation of large language models (LLMs) in low-resource languages like Persian has received less attention than in high-resource languages like English. Existing evaluation approaches for Persian LLMs generally lack comprehensive frameworks, limiting their ability to assess models' performance over a wide range of tasks requiring considerable cultural and contextual knowledge, as well as a deeper understanding of Persian literature and style. This paper first aims to fill this gap by providing two new benchmarks, PeKA and PK-BETS, on topics such as history, literature, and cultural knowledge, as well as challenging the present state-of-the-art models' abilities in a variety of Persian language comprehension tasks. These datasets are meant to reduce data contamination while providing an accurate assessment of Persian LLMs. The second aim of this paper is the general evaluation of LLMs across the current Persian benchmarks to provide a comprehensive performance overview. By offering a structured evaluation methodology, we hope to promote the examination of LLMs in the Persian language."
}