@inproceedings{janarthan-etal-2026-transformers,
    title = "When transformers learn ``impossible'' languages, what do they learn?",
    author = "Janarthan, Ram  and
      Haley, Coleman  and
      Goldwater, Sharon",
    editor = "Bonial, Claire  and
      Berzak, Yevgeni",
    booktitle = "Proceedings of the 30th Conference on Computational Natural Language Learning",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.conll-main.24/",
    pages = "421--434",
    ISBN = "979-8-89176-410-1",
    abstract = "Recent work suggests that transformer language models show a bias towards human languages over unnatural ({''}impossible'') languages argued to be unacquirable by humans. However, this literature has largely based these claims on differences in sample efficiency and test-set perplexity, rather than on direct evaluations of the linguistic capacities that could plausibly explain non-attestation in human languages. We evaluate two theoretically motivated linking hypotheses: impossibility arising from deficiencies in grammatical sensitivity or generative production. Using GPT-2 style models trained on perturbed ``impossible'' variants of English, we measure sensitivity to grammaticality using BLiMP minimal pairs, finding that model performance exhibits only gradual degradation, mediated by the language{'}s information locality. In contrast, these models exhibited pronounced failures in generation, producing substantially fewer high-quality sentences at longer lengths. Together, these results suggest generative deficiency and transmission failures as a plausible linking hypothesis between language model behaviour and non-attestation of impossible languages."
}