@inproceedings{capone-etal-2024-babies,
    title = "{B}a{BIE}s: A Benchmark for the Linguistic Evaluation of {I}talian Baby Language Models",
    author = "Capone, Luca  and
      Suozzi, Alice  and
      Lebani, Gianluca  and
      Lenci, Alessandro",
    editor = "Dell'Orletta, Felice  and
      Lenci, Alessandro  and
      Montemagni, Simonetta  and
      Sprugnoli, Rachele",
    booktitle = "Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)",
    month = dec,
    year = "2024",
    address = "Pisa, Italy",
    publisher = "CEUR Workshop Proceedings",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.clicit-1.20/",
    pages = "157--170",
    ISBN = "979-12-210-7060-6",
    abstract = "The possibility of comparing the linguistic competence of Language Models (LMs) to that of children has gained growing attention lately, raising the need for effective tools for evaluating both the former and the latter. To this purpose, we developed a resource for the linguistic evaluation of BabyLMs, which are LMs trained on datasets that comparable to the linguistic stimulus received by children. This resource adapts four standardized tests for the evaluation of linguistic skills of Italian-speaking children (BVL, TROG-2, TCGB-2 and Peabody). To verify the effectiveness of our benchmark, we administered it to Minerva, a LLM pretrained from scratch on Italian. Our results indicate that Minerva struggles to master certain linguistic aspects, achieving an age-equivalent score of 4 years, and that the type of task administered affects the model{'}s performance."
}