@inproceedings{vintar-etal-2026-charting,
    title = "Charting the {E}uropean {LLM} Benchmarking Landscape: A New Taxonomy and Registry",
    author = "Vintar, Spela  and
      Brglez, Mojca  and
      Kuzman Punger{\v{s}}ek, Taja  and
      Ljube{\v{s}}i{\'c}, Nikola",
    editor = "Montejo-Raez, Arturo  and
      Grisot, Cristina  and
      Blochowiak, Joanna  and
      Ljube{\v{s}}i{\'c}, Nikola  and
      Battaner, Elena  and
      Rigau, German",
    booktitle = "Proceedings of Shaping Multilingual, Multimodal {AI} for the Social Sciences and Humanities ({LLM}s4{SSH}) @ {LREC} 2026",
    month = may,
    year = "2026",
    address = "Palma de Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/ingest-evalita/2026.llms4ssh-1.22/",
    doi = "10.63317/4kixe3c9zmde",
    pages = "205--217",
    abstract = "While new benchmarks for large language models (LLMs) are being developed continuously to catch up with the growing capabilities of new models and AI in general, using and evaluating LLMs in non-English languages remains a poorly-charted landscape. We give a concise overview of recent developments in LLM benchmarking, and then propose a new taxonomy for the categorization of benchmarks that is tailored to multilingual or non-English use scenarios. We further propose a registry of benchmarks implementing the new categorization and documenting benchmarks with a rich set of metadescriptors. While still at a pilot stage, such a registry can lead to a more coordinated development of benchmarks for European languages. We conclude with a review of current trends and advocate for a higher language and culture sensitivity of evaluation methods."
}