@inproceedings{chen-etal-2026-robustness,
    title = "Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction",
    author = "Chen, Yulin  and
      Li, Haoran  and
      Sui, Yuan  and
      Liu, Yue  and
      He, Yufei  and
      Bai, Xiaoling  and
      Fei, Chi  and
      Yabo, Li  and
      Ma, Haozhe  and
      Song, Yangqiu  and
      Hooi, Bryan",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.61/",
    pages = "1194--1215",
    ISBN = "979-8-89176-395-1",
    abstract = "Prompt injection attacks manipulate large language models (LLMs) by misleading them to deviate from the original input instructions and execute maliciously injected instructions, because of their instruction-following capabilities and inability to distinguish between the original input instructions and maliciously injected instructions. Currently, various prompt injection defense methods have been proposed, including prompt-engineering-based approaches and fine-tuning methods. Most of these methods instruct the model to follow the original input instructions, suppressing its inherent tendencies to follow the injected instructions. However, experimental results reveal that suppressing the model{'}s instruction-following tendencies is challenging. After analyzing successful attack cases, we find that the LLMs can correctly reference the instructions they are executing in some cases. Motivated by this finding, we propose a defense method that leverages LLMs' instruction-following abilities rather than suppressing them. Our approach prompts LLMs to generate responses that include both the answers and their corresponding instruction references. Based on these references, we filter out answers whose references are not to the original input instructions. We conduct comprehensive experiments to evaluate the effectiveness of our proposed method. The results show that our approach outperforms prompt-engineering-based baselines and is comparable to fine-tuning methods, reducing the ASR to nearly 0{\%} in some scenarios. Moreover, our approach has minimal impact on overall utility."
}