@inproceedings{chiang-etal-2026-shanks,
    title = "Shanks: Simultaneous Hearing and Thinking for Spoken Language Models",
    author = "Chiang, Cheng-Han  and
      Wang, Xiaofei  and
      Li, Linjie  and
      Lin, Chung-Ching  and
      Lin, Kevin  and
      Liu, Shujie  and
      Wang, Zhendong  and
      Yang, Zhengyuan  and
      Lee, Hung-yi  and
      Wang, Lijuan",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.404/",
    pages = "8951--8972",
    ISBN = "979-8-89176-390-6",
    abstract = "Current large language models (LLMs) and spoken language models (SLMs) begin thinking and taking actions only after the user has finished their turn. This prevents the model from interacting with the user during the user{'}s turn and can lead to high response latency when the model is thinking. To address this issue, we draw inspiration from the ``think while listening'' behavior of humans. In this paper, we propose SHANKS, a general inference framework that enables SLMs to generate unspoken chain-of-thought reasoning while listening to user input. SHANKS streams input speech in fixed-duration chunks and, as soon as a chunk is received, generates unspoken reasoning based on all previous speech and reasoning, while the user continues speaking. SHANKS uses unspoken reasoning to determine whether to interrupt the user and make tool calls to complete the task. We demonstrate that SHANKS enhances real-time user{--}SLM interaction in two scenarios: (1) SHANKS can listen to the user{'}s speech and interrupt when the user makes a mistake. (2) In a tool-augmented dialogue scenario, SHANKS can complete 56.9{\%} of the tool calls before the user ends their turn. Overall, SHANKS is a step toward models that keep thinking throughout the conversation, not only after a turn ends. Demos can be found on the project page: https://d223302.github.io/SHANKS/."
}