@inproceedings{vejlgaard-holm-etal-2025-danoliteracy,
    title = "Danoliteracy of Generative Large Language Models",
    author = "Vejlgaard Holm, S{\o}ren  and
      Hansen, Lars Kai  and
      Nielsen, Martin Carsten",
    editor = "Johansson, Richard  and
      Stymne, Sara",
    booktitle = "Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)",
    month = mar,
    year = "2025",
    address = "Tallinn, Estonia",
    publisher = "University of Tartu Library",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.nodalida-1.78/",
    pages = "785--800",
    ISBN = "978-9908-53-109-0",
    abstract = "The language technology moonshot moment of Generative Large Language Models (GLLMs) was not limited to English: These models brought a surge of technological applications, investments, and hype to low-resource languages as well. However, the capabilities of these models in languages such as Danish were, until recently, difficult to verify beyond qualitative demonstrations due to a lack of applicable evaluation corpora. We present a GLLM benchmark to evaluate Danoliteracy, a measure of Danish language and cultural competency across eight diverse scenarios such as Danish citizenship tests and abstractive social media question answering. This limited-size benchmark was found to produce a robust ranking that correlates to human feedback at $\rho \sim 0.8$ with GPT-4 and Claude Opus models achieving the highest rankings. Analyzing these model results across scenarios, we find one strong underlying factor explaining 95{\%} of scenario performance variance for GLLMs in Danish, suggesting a $g$ factor of model consistency in language adaptation."
}