@inproceedings{boglioni-etal-2026-generalisation,
    title = "Do Generalisation Results Generalise?",
    author = "Boglioni, Matteo  and
      Sgobbi, Andrea  and
      Tavernini, Gabriel  and
      Rita, Francesco  and
      Mosbach, Marius  and
      Pimentel, Tiago",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1284/",
    pages = "25763--25783",
    ISBN = "979-8-89176-395-1",
    abstract = "A large language model{'}s (LLM{'}s) out-of-distribution (OOD) generalisation is crucial to its deployment. Previous work assessing LLMs' generalisation performance, however, typically focuses on a single out-of-distribution dataset. This approach may fail to precisely evaluate the capabilities of the model, as the data shifts encountered during deployment are much more diverse. In this work, we investigate whether OOD generalisation results generalise. More specifically, we evaluate a model{'}s performance across multiple OOD testsets throughout a finetuning run; we then evaluate the partial correlation of performances across these testsets, regressing out in-domain performance. This allows us to assess how correlated are generalisation performances once in-domain performance is controlled for. Analysing OLMo, OPT and SmolLM, we observe no overarching trend in generalisation results: the existence of a positive or negative correlation between any two OOD testsets depends strongly on the specific choice of model analysed."
}