@inproceedings{chi-etal-2025-modeling,
    title = "{M}ode{L}ing: A Novel Dataset for Testing Linguistic Reasoning in Language Models",
    author = "Chi, Nathan Andrew  and
      Malchev, Teodor  and
      Kong, Riley  and
      Chi, Ryan Andrew  and
      Huang, Lucas  and
      Chi, Ethan A  and
      McCoy, R. Thomas  and
      Radev, Dragomir",
    editor = "Ojha, Atul Kr.  and
      Liu, Chao-hong  and
      Vylomova, Ekaterina  and
      Pirinen, Flammie  and
      Washington, Jonathan  and
      Oco, Nathaniel  and
      Zhao, Xiaobing",
    booktitle = "Proceedings of the Eighth Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2025)",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico, U.S.A.",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.loresmt-1.10/",
    pages = "105--114",
    ISBN = "979-8-89176-230-5",
    abstract = "We introduce ModeLing, a novel benchmark of Linguistics Olympiad-style puzzles which tests few-shot reasoning in AI systems. Solving these puzzles necessitates inferring aspects of a language`s grammatical structure from a small number of examples. Such puzzles provide a natural testbed for language models, as they require compositional generalization and few-shot inductive reasoning. Consisting solely of new puzzles written specifically for this work, ModeLing has no risk of appearing in the training data of existing AI systems: this ameliorates the risk of data leakage, a potential confounder for many prior evaluations of reasoning. Evaluating several large open source language models and GPT on our benchmark, we observe non-negligible accuracy, demonstrating few-shot emergent reasoning ability which cannot merely be attributed to shallow memorization. However, imperfect model performance suggests that ModeLing can be used to measure further progress in linguistic reasoning."
}