@inproceedings{basar-bisazza-2026-morphology,
    title = "A Morphology-Aware Evaluation of {T}urkish Syntax in Large Language Models",
    author = "Ba{\c{s}}ar, Ezgi  and
      Bisazza, Arianna",
    editor = {Oflazer, Kemal  and
      K{\"o}ksal, Abdullatif  and
      Varol, Onur},
    booktitle = "Proceedings of the Second Workshop Natural Language Processing for {T}urkic Languages ({SIGTURK} 2026)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/manual-author-scripts/2026.sigturk-1.9/",
    pages = "95--102",
    ISBN = "979-8-89176-370-8",
    abstract = "Minimal pair benchmarks have become a common approach for evaluating the syntactic knowledge of language models (LMs). However, the creation of such benchmarks often overlooks language-specific confounders that may affect model performance, particularly in the case of morphologically rich languages. In this paper, we investigate how surface-level factors such as morpheme count, subword count, and sentence length influence the performance of LMs on a Turkish benchmark of linguistic minimal pairs. We further analyze whether a tokenizer{'}s degree of alignment with morphological boundaries can serve as a proxy for model performance. Finally, we test whether the distribution of morphemes in a minimal pair benchmark can skew model performance. Our results show that while surface factors have limited predictive power, they might still serve as a systematic source of bias. Moreover, we find that morphological alignment can roughly correspond to model performance, and morpheme-level imbalances in the benchmark may have a significant influence on results."
}