@inproceedings{ligeti-nagy-etal-2025-hugme,
    title = "{H}u{GME}: A benchmark system for evaluating {H}ungarian generative {LLM}s",
    author = "Ligeti-Nagy, No{\'e}mi  and
      Madarasz, Gabor  and
      Foldesi, Flora  and
      Lengyel, Mariann  and
      Osvath, Matyas  and
      Sarossy, Bence  and
      Varga, Kristof  and
      Yang, Gy{\H{o}}z{\H{o}} Zijian  and
      H{\'e}ja, Enik{\H{o}}  and
      V{\'a}radi, Tam{\'a}s  and
      Pr{\'o}sz{\'e}ky, G{\'a}bor",
    editor = "Dhole, Kaustubh  and
      Clinciu, Miruna",
    booktitle = "Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM{\texttwosuperior})",
    month = jul,
    year = "2025",
    address = "Vienna, Austria and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-08/2025.gem-1.32/",
    pages = "385--403",
    ISBN = "979-8-89176-261-9",
    abstract = "In this study, we introduce the Hungarian Generative Model Evaluation (HuGME) benchmark, a new framework designed to assess the linguistic proficiency of large language models (LLMs) in Hungarian. HuGME evaluates models across a diverse set of linguistic and reasoning skills, including bias, toxicity, faithfulness, relevance, summarization, prompt alignment, readability, spelling, grammaticality, and domain-specific knowledge through tasks like TruthfulQA and MMLU. We applied HuGME to a range of Hungarian LLMs, including those developed in-house as well as several publicly available models that claim Hungarian language proficiency. This paper presents the comparative results of these evaluations, shedding light on the capabilities of current LLMs in processing the Hungarian language. Through our analysis, we aim to both showcase the current state of Hungarian linguistic processing in LLMs and provide a foundational resource for future advancements in the field."
}