@inproceedings{attanasio-etal-2024-calamita,
    title = "{CALAMITA}: Challenge the Abilities of {LA}nguage Models in {ITA}lian",
    author = "Attanasio, Giuseppe  and
      Basile, Pierpaolo  and
      Borazio, Federico  and
      Croce, Danilo  and
      Francis, Maria  and
      Gili, Jacopo  and
      Musacchio, Elio  and
      Nissim, Malvina  and
      Patti, Viviana  and
      Rinaldi, Matteo  and
      Scalena, Daniel",
    editor = "Dell'Orletta, Felice  and
      Lenci, Alessandro  and
      Montemagni, Simonetta  and
      Sprugnoli, Rachele",
    booktitle = "Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)",
    month = dec,
    year = "2024",
    address = "Pisa, Italy",
    publisher = "CEUR Workshop Proceedings",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/2024.clicit-1.116/",
    pages = "1054--1063",
    ISBN = "979-12-210-7060-6",
    abstract = "The rapid development of Large Language Models (LLMs) has called for robust benchmarks to assess their abilities, track progress, and compare iterations. While existing benchmarks provide extensive evaluations across diverse tasks, they predominantly focus on English, leaving other languages underserved. For Italian, the EVALITA campaigns have provided a long-standing tradition of classification-focused shared tasks. However, their scope does not fully align with the nuanced evaluation required for modern LLMs. To address this gap, we introduce {\textquotedblleft}Challenge the Abilities of LAnguage Models in ITAlian{\textquotedblright} (CALAMITA), a collaborative effort to create a dynamic and growing benchmark tailored to Italian. CALAMITA emphasizes diversity in task design to test a wide range of LLM capabilities through resources natively developed in Italian by the community. This initiative includes a shared platform, live leaderboard, and centralized evaluation framework. This paper outlines the collaborative process, initial challenges, and evaluation framework of CALAMITA."
}