@inproceedings{farsi-etal-2025-melac,
    title = "{MELAC}: Massive Evaluation of Large Language Models with Alignment of Culture in {P}ersian Language",
    author = "Farsi, Farhan  and
      Aghababaloo, Farnaz  and
      Motlagh, Shahriar Shariati  and
      Ghofrani, Parsa  and
      SadraeiJavaheri, MohammadAli  and
      Bali, Shayan  and
      Shabani, Amir Hossein  and
      Bijary, Farbod  and
      Zamaninejad, Ghazal  and
      Salehoof, AmirMohammad  and
      Momtazi, Saeedeh",
    editor = "Inui, Kentaro  and
      Sakti, Sakriani  and
      Wang, Haofen  and
      Wong, Derek F.  and
      Bhattacharyya, Pushpak  and
      Banerjee, Biplab  and
      Ekbal, Asif  and
      Chakraborty, Tanmoy  and
      Singh, Dhirendra Pratap",
    booktitle = "Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "The Asian Federation of Natural Language Processing and The Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.ijcnlp-long.105/",
    pages = "1933--1950",
    ISBN = "979-8-89176-298-5",
    abstract = "As large language models (LLMs) become increasingly embedded in our daily lives, evaluating their quality and reliability across diverse contexts has become essential. While comprehensive benchmarks exist for assessing LLM performance in English, there remains a significant gap in evaluation resources for other languages. Moreover, because most LLMs are trained primarily on data rooted in European and American cultures, they often lack familiarity with non-Western cultural contexts. To address this limitation, our study focuses on the Persian language and Iranian culture. We introduce 19 new evaluation datasets specifically designed to assess LLMs on topics such as Iranian law, Persian grammar, Persian idioms, and university entrance exams. Using these datasets, we benchmarked 41 prominent LLMs, aiming to bridge the existing cultural and linguistic evaluation gap in the field. The evaluation results are publicly available on our live leaderboard: https://huggingface.co/spaces/opll-org/Open-Persian-LLM-Leaderboard"
}