TRACE-it: Testing Relative clAuses Comprehension through Entailment in ITalian: A CALAMITA Challenge
Abstract
Introduced in the context of CALAMITA 2024, TRACE-it (Testing Relative clAuses Comprehension through Entailment in ITalian) is a benchmark designed to evaluate the ability of Large Language Models (LLMs) to comprehend a specific type of complex syntactic construction in Italian: object relative clauses. In this report, we outline the theoretical framework that informed the creation of the dataset and provide a comprehensive overview of the linguistic materials used.- Anthology ID:
- 2024.clicit-1.119
- Volume:
- Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)
- Month:
- December
- Year:
- 2024
- Address:
- Pisa, Italy
- Editors:
- Felice Dell'Orletta, Alessandro Lenci, Simonetta Montemagni, Rachele Sprugnoli
- Venue:
- CLiC-it
- SIG:
- Publisher:
- CEUR Workshop Proceedings
- Note:
- Pages:
- 1082–1088
- Language:
- URL:
- https://preview.aclanthology.org/more-markup/2024.clicit-1.119/
- DOI:
- Cite (ACL):
- Dominique Brunato. 2024. TRACE-it: Testing Relative clAuses Comprehension through Entailment in ITalian: A CALAMITA Challenge. In Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), pages 1082–1088, Pisa, Italy. CEUR Workshop Proceedings.
- Cite (Informal):
- TRACE-it: Testing Relative clAuses Comprehension through Entailment in ITalian: A CALAMITA Challenge (Brunato, CLiC-it 2024)
- PDF:
- https://preview.aclanthology.org/more-markup/2024.clicit-1.119.pdf