@inproceedings{rodkin-etal-2026-beyond,
    title = "Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling",
    author = "Rodkin, Ivan  and
      Orel, Daniil  and
      Smirnov, Konstantin  and
      Bolatov, Arman  and
      Elbouardi, Bilal  and
      Hassan, Besher  and
      Kuratov, Yuri  and
      Bulatov, Aydar  and
      Nakov, Preslav  and
      Baldwin, Timothy  and
      Shelmanov, Artem  and
      Burtsev, Mikhail",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2103/",
    pages = "42385--42404",
    ISBN = "979-8-89176-395-1",
    abstract = "Reasoning is a core capability of large language models (LLMs), yet how multi-step reasoning is learned and executed remains unclear. We study this question in a controlled cellular-automata (1dCA) framework that excludes memorization by using disjoint training and test rules. Given a short state sequence, the model is required to infer the hidden local rule and then chain it to predict multiple future steps. Our evaluation shows that LLMs largely fail to reliably solve a natural-language proxy of the proposed task. We find that most neural architectures trained from scratch can learn rule inference and achieve high next-step accuracy, but performance drops sharply as the required number of intermediate reasoning steps increases. Experiments show that increasing model depth is crucial, and extending effective depth via recurrence, memory, or test-time compute improves results but remains bounded. Code is available on github: https://github.com/RodkinIvan/associative-recurrent-memory-transformer/tree/ACT."
}