TRACE-it: Testing Relative clAuses Comprehension through Entailment in ITalian: A CALAMITA Challenge

Dominique Brunato


Abstract
Introduced in the context of CALAMITA 2024, TRACE-it (Testing Relative clAuses Comprehension through Entailment in ITalian) is a benchmark designed to evaluate the ability of Large Language Models (LLMs) to comprehend a specific type of complex syntactic construction in Italian: object relative clauses. In this report, we outline the theoretical framework that informed the creation of the dataset and provide a comprehensive overview of the linguistic materials used.
Anthology ID:
2024.clicit-1.119
Volume:
Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)
Month:
December
Year:
2024
Address:
Pisa, Italy
Editors:
Felice Dell'Orletta, Alessandro Lenci, Simonetta Montemagni, Rachele Sprugnoli
Venue:
CLiC-it
SIG:
Publisher:
CEUR Workshop Proceedings
Note:
Pages:
1082–1088
Language:
URL:
https://preview.aclanthology.org/fix-sig-urls/2024.clicit-1.119/
DOI:
Bibkey:
Cite (ACL):
Dominique Brunato. 2024. TRACE-it: Testing Relative clAuses Comprehension through Entailment in ITalian: A CALAMITA Challenge. In Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), pages 1082–1088, Pisa, Italy. CEUR Workshop Proceedings.
Cite (Informal):
TRACE-it: Testing Relative clAuses Comprehension through Entailment in ITalian: A CALAMITA Challenge (Brunato, CLiC-it 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/fix-sig-urls/2024.clicit-1.119.pdf