Yan Lu
2026
CSPO: Alleviating Reward Ambiguity for Structured Table-to-LaTeX Generation
Yunfan Yang | Cuiling Lan | Jitao Sang | Yan Lu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yunfan Yang | Cuiling Lan | Jitao Sang | Yan Lu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Tables contain rich structured information, yet when stored as images their contents remain "locked" within pixels. Converting table images into LaTeX code enables faithful digitization and reuse, but current multimodal large language models (MLLMs) often fail to preserve structural, style, or content fidelity. Conventional post-training with reinforcement learning (RL) typically relies on a single aggregated reward, leading to reward ambiguity that conflates multiple behavioral aspects and hinders effective optimization. We propose Component-Specific Policy Optimization (CSPO), an RL framework that disentangles optimization across LaTeX tables components—structure, style, and content. In particular, CSPO assigns component-specific rewards and backpropagates each signal only through the tokens relevant to its component, alleviating reward ambiguity and enabling targeted component-wise optimization. To comprehensively assess performance, we introduce a set of hierarchical evaluation metrics. Extensive experiments demonstrate the effectiveness of CSPO, underscoring the importance of component-specific optimization for reliable structured generation.
Closing the Modality Reasoning Gap for Speech Large Language Models
Chaoren Wang | Heng Lu | Xueyao Zhang | Shujie Liu | Yan Lu | Jinyu Li | Zhizheng Wu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Chaoren Wang | Heng Lu | Xueyao Zhang | Shujie Liu | Yan Lu | Jinyu Li | Zhizheng Wu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Although Speech Large Language Models have achieved notable progress, a substantial modality reasoning gap remains: their reasoning performance on speech inputs is markedly weaker than on text. This gap could be associated with representational drift across Transformer layers and behavior deviations in long-chain reasoning. To address this issue, we introduce TARS, a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories through an asymmetric reward design. The framework employs two dense and complementary signals: representation alignment, which measures layer-wise hidden-state similarity between speech- and text-conditioned trajectories, and behavior alignment, which evaluates semantic consistency between generated outputs and reference text completions. Experiments on challenging reasoning benchmarks, including MMSU and OBQA, show that our approach significantly narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs.
SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation
Hui Wang | Jinghua Zhao | Yifan Yang | Shujie Liu | Junyang Chen | Yanzhe Zhang | Shiwan Zhao | Jinyu Li | Jiaming Zhou | Haoqin Sun | Yan Lu | Yong Qin
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Hui Wang | Jinghua Zhao | Yifan Yang | Shujie Liu | Junyang Chen | Yanzhe Zhang | Shiwan Zhao | Jinyu Li | Jiaming Zhou | Haoqin Sun | Yan Lu | Yong Qin
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Generative speech technologies are progressing rapidly, but evaluating the perceptual quality of synthetic speech remains a core challenge. Existing methods typically rely on scalar scores or binary decisions, which lack interpretability and generalization across tasks and languages. We present SpeechLLM-as-Judges, a new paradigm for enabling large language models (LLMs) to conduct structured and explanation-based speech quality evaluation. To support this direction, we introduce SpeechEval, a large-scale dataset containing 32,207 multilingual speech clips and 128,754 annotations spanning four tasks: quality assessment, pairwise comparison, improvement suggestion, and deepfake detection. Based on this resource, we develop SQ-LLM, a speech-quality-aware LLM trained with chain-of-thought reasoning and reward optimization to improve capability. Experimental results show that SQ-LLM delivers strong performance across tasks and languages, revealing the potential of this paradigm for advancing speech quality evaluation. The relevant code, models, and data are publicly available at https://github.com/NKU-HLT/SpeechLLM-as-Judges.
InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training
Ziyun Zhang | Zezhou Wang | Xiaoyi Zhang | Zongyu Guo | Jiahao Li | Bin Li | Yan Lu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Ziyun Zhang | Zezhou Wang | Xiaoyi Zhang | Zongyu Guo | Jiahao Li | Bin Li | Yan Lu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
GUI agents that interact with graphical interfaces on behalf of users are a promising direction for practical AI assistants, yet training them is hindered by scarce suitable environments. We present InfiniteWeb, a system that automatically generates functional web environments at scale for GUI agent training. While LLMs perform well on generating a single webpage, building a realistic and functional website with many interconnected pages faces challenges. We address these challenges through unified specification, task-centric test-driven development, and combining website seed variation with reference design images. Our system also generates verifiable task evaluators enabling dense reward signals for reinforcement learning. Experiments show that our system surpasses commercial coding agents at realistic website construction, and GUI agents trained on our generated environments achieve significant performance improvements on OSWorld and Online-Mind2Web, demonstrating the effectiveness of the proposed system.
From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation
Zezhou Wang | Ziyun Zhang | Xiaoyi Zhang | Zhuzhong Qian | Yan Lu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Zezhou Wang | Ziyun Zhang | Xiaoyi Zhang | Zhuzhong Qian | Yan Lu
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Vision-language models are increasingly deployed as computer-use agents (CUAs) that operate desktops and browsers. Top-performing CUAs are framework-based systems that decompose planning and execution, while end-to-end screenshot-to-action policies are easier to deploy but lag behind on benchmarks such as OSWorld-Verified. GUI datasets like OSWorld pose two bottlenecks: they expose only a few hundred interactive, verifiable tasks and environments, and expert trajectories must be gathered by interacting with these environments, making such data hard to scale. We therefore ask how reinforcement learning from verifiable rewards (RLVR) can best exploit a small pool of exist expert trajectories to train end-to-end policies. Na"ively mixing these off-policy traces into on-policy RLVR is brittle: even after format conversion, expert trajectories exhibit structural mismatch and distribution shift from the learner. We propose BEPA (Bi-Level Expert-to-Policy Assimilation), which turns static expert traces into policy-aligned guidance via self-rolled reachable trajectories under the base policy (LEVEL-1) and a per-task, dynamically updated cache used in RLVR (LEVEL-2). On OSWorld-Verified, BEPA improves UITARS1.5-7B success from 22.87% to 32.13% and raises a held-out split from 5.74% to 10.30%, with consistent gains on MMBench-GUI and Online-Mind2Web. Our code and data are available at an anonymous repository: https://anonymous.4open.science/r/ACL_BEPA.
2025
TrInk: Ink Generation with Transformer Network
Zezhong Jin | Shubhang Desai | Xu Chen | Biyi Fang | Zhuoyi Huang | Zhe Li | Chong-Xin Gan | Xiao Tu | Man-Wai Mak | Yan Lu | Shujie Liu
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Zezhong Jin | Shubhang Desai | Xu Chen | Biyi Fang | Zhuoyi Huang | Zhe Li | Chong-Xin Gan | Xiao Tu | Man-Wai Mak | Yan Lu | Shujie Liu
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
In this paper, we propose TrInk, a Transformer-based model for ink generation, which effectively captures global dependencies. To better facilitate the alignment between the input text and generated stroke points, we introduce scaled positional embeddings and a Gaussian memory mask in the cross-attention module. Additionally, we design both subjective and objective evaluation pipelines to comprehensively assess the legibility and style consistency of the generated handwriting. Experiments demonstrate that our Transformer-based model achieves a 35.56% reduction in character error rate (CER) and an 29.66% reduction in word error rate (WER) on the IAM-OnDB dataset compared to previous methods. We provide an demo page with handwriting samples from TrInk and baseline models at: https://akahello-a11y.github.io/trink-demo/
SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training
Wenxi Chen | Ziyang Ma | Ruiqi Yan | Yuzhe Liang | Xiquan Li | Ruiyang Xu | Zhikang Niu | Yanqiao Zhu | Yifan Yang | Zhanxun Liu | Kai Yu | Yuxuan Hu | Jinyu Li | Yan Lu | Shujie Liu | Xie Chen
Findings of the Association for Computational Linguistics: ACL 2025
Wenxi Chen | Ziyang Ma | Ruiqi Yan | Yuzhe Liang | Xiquan Li | Ruiyang Xu | Zhikang Niu | Yanqiao Zhu | Yifan Yang | Zhanxun Liu | Kai Yu | Yuxuan Hu | Jinyu Li | Yan Lu | Shujie Liu | Xie Chen
Findings of the Association for Computational Linguistics: ACL 2025
Recent advancements highlight the potential of end-to-end real-time spoken dialogue systems, showcasing their low latency and high quality. In this paper, we introduce SLAM-Omni, a timbre-controllable, end-to-end voice interaction system with single-stage training. SLAM-Omni achieves zero-shot timbre control by modeling spoken language with semantic tokens and decoupling speaker information to a vocoder. By predicting grouped speech semantic tokens at each step, our method significantly reduces the sequence length of audio tokens, accelerating both training and inference. Additionally, we propose historical text prompting to compress dialogue history, facilitating efficient multi-round interactions. Comprehensive evaluations reveal that SLAM-Omni outperforms prior models of similar scale, requiring only 15 hours of training on 4 GPUs with limited data. Notably, it is the first spoken dialogue system to achieve competitive performance with a single-stage training approach, eliminating the need for pre-training on TTS or ASR tasks. Further experiments validate its multilingual and multi-turn dialogue capabilities on larger datasets.
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis
Xinyi Liu | Xiaoyi Zhang | Ziyun Zhang | Yan Lu
Findings of the Association for Computational Linguistics: ACL 2025
Xinyi Liu | Xiaoyi Zhang | Ziyun Zhang | Yan Lu
Findings of the Association for Computational Linguistics: ACL 2025
Recent advancements in Large Vision-Language Models are accelerating the development of Graphical User Interface (GUI) agents that utilize human-like vision perception capabilities to enhance productivity on digital devices. Compared to approaches predicated on GUI metadata, which are platform-dependent and vulnerable to implementation variations, vision-based approaches offer broader applicability.In this vision-based paradigm, the GUI instruction grounding, which maps user instruction to the location of corresponding element on the given screenshot, remains a critical challenge, particularly due to limited public training dataset and resource-intensive manual instruction data annotation.In this paper, we delve into unexplored challenges in this task including element-to-screen ratio, unbalanced element type, and implicit instruction. To address these challenges, we introduce a large-scale data synthesis pipeline UI-E2I-Synth for generating varying complex instruction datasets using GPT-4o instead of human annotators. Furthermore, we propose a new GUI instruction grounding benchmark UI-I2E-Bench, which is designed to address the limitations of existing benchmarks by incorporating diverse annotation aspects.Our model, trained on the synthesized data, achieves superior performance in GUI instruction grounding, demonstrating the advancements of proposed data synthesis pipeline.The proposed benchmark, accompanied by extensive analyses, provides practical insights for future research in this domain. We will release our dataset and benchmark to facilitate further development of GUI instruction grounding community.
2024
UDAA: An Unsupervised Domain Adaptation Adversarial Learning Framework for Zero-Resource Cross-Domain Named Entity Recognition
Baofeng Li | Jianguo Tang | Yu Qin | Yuelou Xu | Yan Lu | Kai Wang | Lei Li | Yanquan Zhou
Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)
Baofeng Li | Jianguo Tang | Yu Qin | Yuelou Xu | Yan Lu | Kai Wang | Lei Li | Yanquan Zhou
Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)
“The zero-resource cross-domain named entity recognition (NER) task aims to perform NER in aspecific domain where labeled data is unavailable. Existing methods primarily focus on transfer-ring NER knowledge from high-resource to zero-resource domains. However, the challenge liesin effectively transferring NER knowledge between domains due to the inherent differences inentity structures across domains. To tackle this challenge, we propose an Unsupervised DomainAdaptation Adversarial (UDAA) framework, which combines the masked language model auxil-iary task with the domain adaptive adversarial network to mitigate inter-domain differences andefficiently facilitate knowledge transfer. Experimental results on CBS, Twitter, and WNUT2016three datasets demonstrate the effectiveness of our framework. Notably, we achieved new state-of-the-art performance on the three datasets. Our code will be released.Introduction”
2012
Entends-tu mes attitudes ? Perception de la prosodie des affects sociaux en chinois Mandarin (Do you hear my attitudes? Perception of Mandarin Chinese social affects’ prosody) [in French]
Yan Lu | Véronique Aubergé | Albert Rilliard
Proceedings of the Joint Conference JEP-TALN-RECITAL 2012, volume 1: JEP
Yan Lu | Véronique Aubergé | Albert Rilliard
Proceedings of the Joint Conference JEP-TALN-RECITAL 2012, volume 1: JEP
Search
Fix author
Co-authors
- Shujie Liu 4
- Jinyu Li 3
- Xiaoyi Zhang 3
- Ziyun Zhang 3
- Zezhou Wang 2
- Yifan Yang 2
- Véronique Aubergé 1
- Xu Chen 1
- Wenxi Chen 1
- Xie Chen 1
- Junyang Chen 1
- Shubhang Desai 1
- Biyi Fang 1
- Chong-Xin Gan 1
- Zongyu Guo 1
- Yuxuan Hu 1
- Zhuoyi Huang 1
- Zezhong Jin 1
- Cuiling Lan 1
- Zhe Li 1
- Xiquan Li 1
- Jiahao Li 1
- Bin Li 1
- Baofeng Li 1
- Lei Li 1
- Yuzhe Liang 1
- Zhanxun Liu 1
- Xinyi Liu 1
- Heng Lu 1
- Ziyang Ma 1
- Man-Wai Mak 1
- Zhikang Niu 1
- Zhuzhong Qian 1
- Yong Qin 1
- Yu Qin 1
- Albert Rilliard 1
- Jitao Sang (桑基韬) 1
- Haoqin Sun 1
- Jianguo Tang 1
- Xiao Tu 1
- Chaoren Wang 1
- Hui Wang 1
- Kai Wang 1
- Zhizheng Wu 1
- Ruiyang Xu 1
- Yuelou Xu 1
- Ruiqi Yan 1
- Yunfan Yang 1
- Kai Yu 1
- Xueyao Zhang 1
- Yanzhe Zhang 1
- Jinghua Zhao 1
- Shiwan Zhao 1
- Jiaming Zhou 1
- Yanquan Zhou 1
- Yanqiao Zhu 1