@inproceedings{attanasio-etal-2024-itaeval-calamita,
    title = "{I}ta{E}val: A {CALAMITA} Challenge",
    author = "Attanasio, Giuseppe  and
      La Quatra, Moreno  and
      Santilli, Andrea  and
      Savoldi, Beatrice",
    editor = "Dell'Orletta, Felice  and
      Lenci, Alessandro  and
      Montemagni, Simonetta  and
      Sprugnoli, Rachele",
    booktitle = "Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)",
    month = dec,
    year = "2024",
    address = "Pisa, Italy",
    publisher = "CEUR Workshop Proceedings",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.clicit-1.117/",
    pages = "1064--1073",
    ISBN = "979-12-210-7060-6",
    abstract = "In recent years, new language models for Italian have been spurring.However, evaluation methodologies for these models have not kept pace, remaining fragmented and often limited to the experimental sections of individual model releases. This paper introduces ItaEval, a multifaceted evaluation suite designed to address this gap. By reviewing recent literature on the evaluation of contemporary language models, we devise three overarching task categories{---}natural language understanding, commonsense and factual knowledge, and bias, fairness, and safety{---}that a contemporary model should be able to address. Next, we collect a set of 18 tasks encompassing existing and new datasets. The so-compiled ItaEval suite provides a standardized, multifaceted framework for evaluating Italian language models, facilitating more rigorous and comparative assessments of model performance. We release code and data at https://rita-nlp.org/sprints/itaeval."
}