@inproceedings{wan-etal-2026-speech,
    title = "Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception",
    author = "Wan, Zhen  and
      Yang, Chao-Han Huck  and
      Tian, Jinchuan  and
      Ye, Hanrong  and
      Pasad, Ankita  and
      Fu, Szu-Wei  and
      Goel, Arushi  and
      Hachiuma, Ryo  and
      Diao, Shizhe  and
      Dhawan, Kunal  and
      Ghosh, Sreyan  and
      Hirota, Yusuke  and
      Chen, Zhehuai  and
      Valle, Rafael  and
      Chu, Chenhui  and
      Watanabe, Shinji  and
      Ginsburg, Boris  and
      Wang, Yu-Chiang Frank",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1997/",
    pages = "43124--43142",
    ISBN = "979-8-89176-390-6",
    abstract = "We introduce a voice-agentic framework that learns one critical omni-understanding skill: knowing when to trust itself versus when to consult external audio perception. Our work is motivated by a crucial yet counterintuitive finding: naively fine-tuning an omni-model on both speech recognition and external sound understanding tasks often degrades performance, as the model can be easily misled by noisy hypotheses. To address this, our framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. This learnable reflection primitive proves effective in preventing the model from being derailed by flawed external candidates. We show that this agentic action mechanism generalizes naturally from speech recognition to complex, multiple-choice audio reasoning. Across the OpenASR leaderboard, which includes seven domain-diverse speech datasets, Speech-Hands consistently outperforms strong baselines by 12.1{\%} WER on the OpenASR benchmark. The model also achieves 77.37{\%} accuracy and high F1 on audio QA decisions, showing robust generalization and reliability across diverse audio question answering datasets. By unifying perception and decision-making, our work offers a practical path toward more reliable and resilient audio intelligence."
}