A large body of research has examined the linguistic abilities of language models (LMs) across various languages. However, conclusive evidence regarding their semantic competence and world knowledge remains limited, especially for low-resource languages. In this study, we explore the semantic competence of Italian BabyLMs, focusing on their sensitivity to semantic violations. To this end, we adapt a minimal pair benchmark targeting semantic violations to evaluate the semantic abilities of BAMBI, a family of small-scale models trained on progressively larger and more complex datasets. We further compare their performance, assessed through accuracy, mean log-likelihood offset, and expected calibration error, with that of three larger Italian LMs. Our findings shed light on this aspect of semantic competence in small-scale models and how this is affected by data scale and training strategies.

The possibility of comparing the linguistic competence of Language Models (LMs) to that of children has gained growing attention lately, raising the need for effective tools for evaluating both the former and the latter. To this purpose, we developed a resource for the linguistic evaluation of BabyLMs, which are LMs trained on datasets that comparable to the linguistic stimulus received by children. This resource adapts four standardized tests for the evaluation of linguistic skills of Italian-speaking children (BVL, TROG-2, TCGB-2 and Peabody). To verify the effectiveness of our benchmark, we administered it to Minerva, a LLM pretrained from scratch on Italian. Our results indicate that Minerva struggles to master certain linguistic aspects, achieving an age-equivalent score of 4 years, and that the type of task administered affects the model’s performance.