@inproceedings{yun-etal-2025-abstract,
    title = "What is an ``Abstract Reasoner''? Revisiting Experiments and Arguments about Large Language Models",
    author = "Yun, Tian  and
      Sun, Chen  and
      Pavlick, Ellie",
    editor = "Boleda, Gemma  and
      Roth, Michael",
    booktitle = "Proceedings of the 29th Conference on Computational Natural Language Learning",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.conll-1.11/",
    pages = "156--168",
    ISBN = "979-8-89176-271-8",
    abstract = "Recent work has argued that large language models (LLMs) are not ``abstract reasoners'', citing their poor zero-shot performance on a variety of challenging tasks as evidence. We revisit these experiments in order to add nuance to the claim. First, we show that while LLMs indeed perform poorly in a zero-shot setting, even tuning a small subset of parameters for input encoding can enable near-perfect performance. However, we also show that this finetuning does not necessarily transfer across datasets. We take this collection of empirical results as an invitation to (re-)open the discussion of what it means to be an ``abstract reasoner'', and why it matters whether LLMs fit the bill."
}