Yazhe Niu
2026
Listen, Pause, and Reason: Toward Perception-Grounded Hybrid Reasoning for Audio Understanding
Jieyi Wang | Yazhe Niu | Dexuan Xu | Zhongyu Wei
Findings of the Association for Computational Linguistics: ACL 2026
Jieyi Wang | Yazhe Niu | Dexuan Xu | Zhongyu Wei
Findings of the Association for Computational Linguistics: ACL 2026
Recent Large Audio Language Models (LALMs) have shown strong capabilities in audio understanding, yet their reasoning remains vulnerable to perceptual errors, especially in noisy and multi-speaker environments. We argue that reliable audio reasoning requires first grounding model’s perception in structured auditory scenes. Motivated by Auditory Scene Analysis, we introduce **PAQA**, a large-scale dataset for **Perception-Aware Question Answering** covering over 300 categories. PAQA adopts a hierarchical decoupling strategy that separates speech from environmental sounds and distinguishes among multiple speakers, providing explicit perceptual supervision for audio reasoning. Building on this, we propose **HyPeR**, a two-stage **Hybrid Perception-Reasoning** framework for perception-grounded audio understanding. In Stage I, the model is fine-tuned on PAQA for cold start to improve perception of acoustic attributes in complex auditory scenes. In Stage II, we further refine its internal reasoning via **Group Relative Policy Optimization (GRPO)**. To support deliberation under acoustic ambiguity, we introduce **PAUSE tokens** for latent computation and a **Perceptual Consistency Reward** to align reasoning rationales with the underlying audio evidence. Extensive ablation studies isolate the effects of the perception-attention mechanism, self-correction module, and pause-based reasoning strategy. Experiments on multiple benchmarks show that HyPeR consistently improves over the base model, including on MMAU-mini (+13.1%), MMAR (+25.5%), and PAQA (+28.2%), while achieving performance comparable to much larger models. Additional analyses of inference latency and computational overhead show that these gains come with acceptable efficiency trade-offs. Overall, our results demonstrate the effectiveness of hybrid perception-grounded reasoning for robust audio understanding.
2025
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
Yuxiang Chai | Siyuan Huang | Yazhe Niu | Han Xiao | Liang Liu | Guozhi Wang | Dingyu Zhang | Shuai Ren | Hongsheng Li
Findings of the Association for Computational Linguistics: ACL 2025
Yuxiang Chai | Siyuan Huang | Yazhe Niu | Han Xiao | Liang Liu | Guozhi Wang | Dingyu Zhang | Shuai Ren | Hongsheng Li
Findings of the Association for Computational Linguistics: ACL 2025
AI agents have drawn increasing attention mostly on their ability to perceive environments, understand tasks, and autonomously achieve goals. To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents which are capable of completing tasks by directly interacting with the graphical user interface (GUI) on mobile devices. AMEX comprises over 104K high-resolution screenshots from popular mobile applications, which are annotated at multiple levels. Unlike existing GUI-related datasets, e.g., Rico, AitW, etc., AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions, and complex natural language instructions with stepwise GUI-action chains. We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets. Additionally, we finetune a baseline model SPHINX Agent and illustrate the effectiveness of AMEX.