@inproceedings{susanto-etal-2025-sea,
    title = "{SEA}-{HELM}: {S}outheast {A}sian Holistic Evaluation of Language Models",
    author = "Susanto, Yosephine  and
      Hulagadri, Adithya Venkatadri  and
      Montalan, Jann Railey  and
      Ngui, Jian Gang  and
      Yong, Xianbin  and
      Leong, Wei Qi  and
      Rengarajan, Hamsawardhini  and
      Limkonchotiwat, Peerat  and
      Mai, Yifan  and
      Tjhi, William Chandra",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.636/",
    pages = "12308--12336",
    ISBN = "979-8-89176-256-5",
    abstract = "With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multiculturalbenchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specificcapabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA)region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far.Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprisingfive core pillars: (1) NLP CLASSICS, (2) LLM-SPECIFICS, (3) SEA LINGUISTICS, (4) SEA CULTURE, (5) SAFETY. SEA-HELMcurrently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models' multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available."
}