Jingyu Xiao
2026
JARVIS or Ultron? A Survey on the Safety and Security Threats of Computer-Using Agents
Ada Chen | Yongjiang Wu | Junyuan Zhang | Jingyu Xiao | Shu Yang | Jen-tse Huang | Kun Wang | Wenxuan Wang | Shuai Wang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Ada Chen | Yongjiang Wu | Junyuan Zhang | Jingyu Xiao | Shu Yang | Jen-tse Huang | Kun Wang | Wenxuan Wang | Shuai Wang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Recently, AI-driven interactions with computing devices have advanced from basic prototype tools to sophisticated, LLM-based systems that emulate human-like operations in graphical user interfaces. We are now witnessing the emergence of Computer-Using Agents (CUAs), capable of autonomously performing tasks such as navigating desktop applications, web pages, and mobile apps. However, as these agents grow in capability, they also introduce novel safety and security risks. Vulnerabilities in LLM-driven reasoning, with the added complexity of integrating multiple software components and multimodal inputs, further complicate the security landscape. In this paper, we present a systematization of knowledge on the safety and security threats of CUAs. We conduct a comprehensive literature review and distill our findings along four research objectives: (i) define the CUA that suits safety analysis; (ii) categorize current safety threats among CUAs; (iii) propose a comprehensive taxonomy of existing defensive strategies; (iv) summarize prevailing benchmarks, datasets, and evaluation metrics used to assess the safety and performance of CUAs. Building on these insights, our work provides future researchers with a structured foundation for exploring unexplored vulnerabilities and offers practitioners actionable guidance in designing and deploying secure Computer-Using Agents.
2025
QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language
Qingsong Zou | Jingyu Xiao | Qing Li | Zhi Yan | Yuhang Wang | Li Xu | Wenxuan Wang | Kuofeng Gao | Ruoyu Li | Yong Jiang
Findings of the Association for Computational Linguistics: ACL 2025
Qingsong Zou | Jingyu Xiao | Qing Li | Zhi Yan | Yuhang Wang | Li Xu | Wenxuan Wang | Kuofeng Gao | Ruoyu Li | Yong Jiang
Findings of the Association for Computational Linguistics: ACL 2025
Recent advances in large language models (LLMs) have demonstrated remarkable potential in the field of natural language processing. Unfortunately, LLMs face significant security and ethical risks. Although techniques such as safety alignment are developed for defense, prior researches reveal the possibility of bypassing such defenses through well-designed jailbreak attacks. In this paper, we propose QueryAttack, a novel framework to examine the generalizability of safety alignment. By treating LLMs as knowledge databases, we translate malicious queries in natural language into structured non-natural query language to bypass the safety alignment mechanisms of LLMs. We conduct extensive experiments on mainstream LLMs, and the results show that QueryAttack not only can achieve high attack success rates (ASRs), but also can jailbreak various defense methods. Furthermore, we tailor a defense method against QueryAttack, which can reduce ASR by up to 64% on GPT-4-1106. Our code is available at https://anonymous.4open.science/r/QueryAttack-334B.
SlideCoder: Layout-aware RAG-enhanced Hierarchical Slide Generation from Design
Wenxin Tang | Jingyu Xiao | Wenxuan Jiang | Xi Xiao | Yuhang Wang | Xuxin Tang | Qing Li | Yuehe Ma | Junliang Liu | Shisong Tang | Michael R. Lyu
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Wenxin Tang | Jingyu Xiao | Wenxuan Jiang | Xi Xiao | Yuhang Wang | Xuxin Tang | Qing Li | Yuehe Ma | Junliang Liu | Shisong Tang | Michael R. Lyu
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Manual slide creation is labor-intensive and requires expert prior knowledge. Existing natural language-based LLM generation methods struggle to capture the visual and structural nuances of slide designs. To address this, we formalize the Reference Image to Slide Generation task and propose Slide2Code, the first benchmark with difficulty-tiered samples based on a novel Slide Complexity Metric. We introduce SlideCoder, a layout-aware, retrieval-augmented framework for generating editable slides from reference images. SlideCoder integrates a Color Gradient-based Segmentation algorithm and a Hierarchical Retrieval-Augmented Generation method to decompose complex tasks and enhance code generation. We also release SlideMaster, a 7B open-source model fine-tuned with improved reverse-engineered data. Experiments show that SlideCoder outperforms state-of-the-art baselines by up to 40.5 points, demonstrating strong performance across layout fidelity, execution accuracy, and visual consistency. Our code is available at https://github.com/vinsontang1/SlideCoder.