@inproceedings{catalan-gris-etal-2026-linguists,
    title = "The Linguist{'}s Lie Detector: Linguistic Knowledge in Large Language Models",
    author = "Catal{\'a}n Gris, Luc{\'i}a  and
      Gerdes, Kim  and
      Lee, John S. Y.",
    editor = "Rehm, Georg  and
      Dietze, Stefan  and
      Dessi, Danilo  and
      Maynard, Diana  and
      Schimmler, Sonja",
    booktitle = "Proceedings of Natural Scientific Language Processing ({NSLP}) @ {LREC} 2026",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/test-year-match/2026.nslp-1.23/",
    doi = "10.63317/2f7zobe6yo7h",
    pages = "235--246",
    abstract = "We present a benchmark and evaluation pipeline for assessing how well large language models (LLMs) handle linguistic knowledge. Starting from a curated subcorpus of 11 syntax-focused articles published in Glossa: A Journal of General Linguistics (2016{--}2026), we design a pipeline that (1) segments article text into sentences, (2) extracts atomic, verifiable statements, and (3) classifies them into linguistic categories (language-specific, typological, theoretical, citation, or structural). Each stage is evaluated against human gold annotations produced by three annotators, with inter-annotator agreement measured via Krippendorff{'}s {\ensuremath{\alpha}} and Cohen{'}s {\ensuremath{\kappa}}. We compare several LLMs on extraction and classification, using BERTScore-style similarity for extraction and macro F1 for classification. Finally, we generate contradictions of the true linguistic statements and test whether LLMs can distinguish true from false claims. On a challenge set of 705 linguistic statements, we compare eight LLMs, with Gemini 3 Flash achieving the highest F1 score of 0.66, indicating that current models possess limited but non-trivial linguistic knowledge."
}