The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective
Zixuan Xia, Haifeng Sun, Jingyu Wang, Qi Qi, Huazheng Wang, Xiaoyuan Fu, Jianxin Liao
Abstract
Prompts, especially high-quality ones, play an invaluable role in assisting large language models (LLMs) to accomplish various natural language processing tasks. However, carefully crafted prompts can also manipulate model behavior. Therefore, the security risks that “prompts themselves face” and those “arising from harmful prompts” cannot be overlooked and we define the Prompt Threat (PT) issues. In this paper, we review the latest attack methods related to prompt threats, focusing on prompt leakage attacks and prompt jailbreak attacks. Additionally, we summarize the experimental setups of these methods and explore the relationship between prompt threats and prompt injection attacks.- Anthology ID:
- 2025.findings-acl.675
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2025
- Month:
- July
- Year:
- 2025
- Address:
- Vienna, Austria
- Editors:
- Wanxiang Che, Joyce Nabende, Ekaterina Shutova, Mohammad Taher Pilehvar
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 12994–13035
- Language:
- URL:
- https://preview.aclanthology.org/landing_page/2025.findings-acl.675/
- DOI:
- Cite (ACL):
- Zixuan Xia, Haifeng Sun, Jingyu Wang, Qi Qi, Huazheng Wang, Xiaoyuan Fu, and Jianxin Liao. 2025. The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12994–13035, Vienna, Austria. Association for Computational Linguistics.
- Cite (Informal):
- The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective (Xia et al., Findings 2025)
- PDF:
- https://preview.aclanthology.org/landing_page/2025.findings-acl.675.pdf