@inproceedings{pensa-etal-2024-gita4calamita,
    title = "{GITA}4{CALAMITA} - Evaluating the Physical Commonsense Understanding of {I}talian {LLM}s in a Multi-layered Approach: A {CALAMITA} Challenge",
    author = "Pensa, Giulia  and
      Azurmendi, Ekhi  and
      Etxaniz, Julen  and
      Altuna, Bego{\~n}a  and
      Gonzalez-Dios, Itziar",
    editor = "Dell'Orletta, Felice  and
      Lenci, Alessandro  and
      Montemagni, Simonetta  and
      Sprugnoli, Rachele",
    booktitle = "Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)",
    month = dec,
    year = "2024",
    address = "Pisa, Italy",
    publisher = "CEUR Workshop Proceedings",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/2024.clicit-1.127/",
    pages = "1153--1160",
    ISBN = "979-12-210-7060-6",
    abstract = "In the context of the CALAMITA Challenge, we investigate the physical commonsense reasoning capabilities of large language models (LLMs) and introduce a methodology to assess their low-level understanding of the physical world. To this end, we use a test set designed to evaluate physical commonsense reasoning in LLMs for the Italian language. We present a tiered dataset, named the Graded Italian Annotated dataset (GITA), which is written and annotated by a professional linguist. This dataset enables us to focus on three distinct levels of commonsense understanding. Our benchmark aims to evaluate three specific tasks: identifying plausible and implausible stories within our dataset, identifying the conflict that generates an implausible story, and identifying the physical states that make a story implausible. We perform these tasks using LLAMA3, and Gemma. Our findings reveal that, although the models may excel at high-level classification tasks, their reasoning is inconsistent and unverifiable, as they fail to capture intermediate evidence."
}