@inproceedings{thorunn-arnardottir-etal-2025-wikiqa,
    title = "{W}iki{QA}-{IS}: Assisted Benchmark Generation and Automated Evaluation of {I}celandic Cultural Knowledge in {LLM}s",
    author = "{\TH}{\'o}runn Arnard{\'o}ttir  and
      El{\'i}as Bjartur Einarsson  and
      Gar{\dh}ar Ingvarsson Juto  and
      {\TH}orvaldur P{\'a}ll Helgason  and
      Hafsteinn Einarsson",
    editor = "Holdt, {\v{S}}pela Arhar  and
      Ilinykh, Nikolai  and
      Scalvini, Barbara  and
      Bruton, Micaella  and
      Debess, Iben Nyholm  and
      Tudor, Crina Madalina",
    booktitle = "Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025)",
    month = mar,
    year = "2025",
    address = "Tallinn, Estonia",
    publisher = "University of Tartu Library, Estonia",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.resourceful-1.13/",
    pages = "64--73",
    ISBN = "978-9908-53-121-2",
    abstract = "This paper presents WikiQA-IS, a novel question-answering dataset focusing on Icelandic culture and history, along with an automated pipeline for dataset generation and evaluation. Leveraging GPT-4 to create questions and answers based on Icelandic Wikipedia articles and news sources, we produced a high-quality corpus of 2,000 question-answer pairs. We introduce an automatic evaluation method using GPT-4o as a judge, which shows strong agreement with human evaluations. Our benchmark reveals varying performances across different language models, with closed-source models generally outperforming open-weights alternatives. This work contributes a resource for evaluating language models' knowledge of Icelandic culture and offers a replicable framework for creating similar datasets in other cultural contexts."
}