@inproceedings{park-etal-2024-open,
    title = "Open {K}o-{LLM} Leaderboard: Evaluating Large Language Models in {K}orean with {K}o-H5 Benchmark",
    author = "Park, Chanjun  and
      Kim, Hyeonwoo  and
      Kim, Dahyun  and
      Cho, SeongHwan  and
      Kim, Sanghoon  and
      Lee, Sukyung  and
      Kim, Yungi  and
      Lee, Hwalsuk",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.acl-long.177/",
    doi = "10.18653/v1/2024.acl-long.177",
    pages = "3220--3234",
    abstract = "This paper introduces the Open Ko-LLM Leaderboard and the Ko-H5 Benchmark as vital tools for evaluating Large Language Models (LLMs) in Korean. Incorporating private test sets while mirroring the English Open LLM Leaderboard, we establish a robust evaluation framework that has been well integrated in the Korean LLM community. We perform data leakage analysis that shows the benefit of private test sets along with a correlation study within the Ko-H5 benchmark and temporal analyses of the Ko-H5 score. Moreover, we present empirical support for the need to expand beyond set benchmarks. We hope the Open Ko-LLM Leaderboard sets precedent for expanding LLM evaluation to foster more linguistic diversity."
}