@inproceedings{farsi-etal-2025-persian,
    title = "{P}ersian in a Court: Benchmarking {VLM}s In {P}ersian Multi-Modal Tasks",
    author = "Farsi, Farhan  and
      Shariati Motlagh, Shahriar  and
      Bali, Shayan  and
      Sabouri, Sadra  and
      Momtazi, Saeedeh",
    editor = "Zhang, Wei Emma  and
      Dai, Xiang  and
      Elliot, Desmond  and
      Fang, Byron  and
      Sim, Mongyuan  and
      Zhuang, Haojie  and
      Chen, Weitong",
    booktitle = "Proceedings of the First Workshop of Evaluation of Multi-Modal Generation",
    month = jan,
    year = "2025",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.evalmg-1.5/",
    pages = "52--56",
    abstract = "This study introduces a novel framework for evaluating Large Language Models (LLMs) and Vision-Language Models (VLMs) in Persian, a low-resource language. We develop comprehensive datasets to assess reasoning, linguistic understanding, and multimodal capabilities. Our datasets include Persian-OCR-QA for optical character recognition, Persian-VQA for visual question answering, Persian world-image puzzle for multimodal integration, Visual-Abstraction-Reasoning for abstract reasoning, and Iran-places for visual knowledge of Iranian figures and locations. We evaluate models like GPT-4o, Claude 3.5 Sonnet, and Llama 3.2 90B Vision, revealing their strengths and weaknesses in processing Persian. This research contributes to inclusive language processing by addressing the unique challenges of low-resource language evaluation."
}