Xin Dai
2026
Revealing the Attention Floating Mechanism in Masked Diffusion Models
Xin Dai | Pengcheng Huang | Zhenghao Liu | Shuo Wang | Yukun Yan | Chaojun Xiao | Yu Gu | Ge Yu | Maosong Sun
Findings of the Association for Computational Linguistics: ACL 2026
Xin Dai | Pengcheng Huang | Zhenghao Liu | Shuo Wang | Yukun Yan | Chaojun Xiao | Yu Gu | Ge Yu | Maosong Sun
Findings of the Association for Computational Linguistics: ACL 2026
Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes and datasets will be available via GitHub.
SAFE-QAQ: End-to-End Slow-Thinking Audio-Text Fraud Detection via Reinforcement Learning
Peidong Wang | Zhiming Ma | Xin Dai | YongKang Liu | Shi Feng | Xiaocui Yang | Wenxing Hu | Zhihao Wang | Mingjun Pan | Li Yuan | Daling Wang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Peidong Wang | Zhiming Ma | Xin Dai | YongKang Liu | Shi Feng | Xiaocui Yang | Wenxing Hu | Zhihao Wang | Mingjun Pan | Li Yuan | Daling Wang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Existing fraud detection methods predominantly rely on transcribed text, suffering from ASR errors and missing crucial acoustic cues like vocal tone and environmental context. This limits their effectiveness against complex deceptive strategies. To address these challenges, we first propose **SAFE-QAQ**, an end-to-end comprehensive framework for audio-based slow-thinking fraud detection. First, the SAFE-QAQ framework eliminates the impact of transcription errors on detection performance. Secondly, we propose rule-based slow-thinking reward mechanisms that systematically guide the system to identify fraud-indicative patterns by accurately capturing fine-grained audio details, through hierarchical reasoning processes. Besides, our framework introduces a dynamic risk assessment framework during live calls, enabling early detection and prevention of fraud. Experiments on the TeleAntiFraud-Bench demonstrate that SAFE-QAQ achieves dramatic improvements over existing methods in multiple key dimensions, including accuracy, inference efficiency, and real-time processing capabilities. Currently deployed and analyzing over 70,000 calls daily, SAFE-QAQ effectively automates complex fraud detection, reducing human workload and financial losses. Code: https://anonymous.4open.science/r/SAFE-QAQ.