@inproceedings{suzuki-etal-2025-superfluous,
    title = "Superfluous Instruction: Vulnerabilities Stemming from Task-Specific Superficial Expressions in Instruction Templates",
    author = "Suzuki, Toma  and
      Sakai, Yusuke  and
      Vasselli, Justin  and
      Kamigaito, Hidetaka  and
      Watanabe, Taro",
    editor = "Zhang, Yuji  and
      Chen, Canyu  and
      Li, Sha  and
      Geva, Mor  and
      Han, Chi  and
      Wang, Xiaozhi  and
      Feng, Shangbin  and
      Gao, Silin  and
      Augenstein, Isabelle  and
      Bansal, Mohit  and
      Li, Manling  and
      Ji, Heng",
    booktitle = "Proceedings of the 3rd Workshop on Towards Knowledgeable Foundation Models (KnowFM)",
    month = aug,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.knowfm-1.12/",
    pages = "140--152",
    ISBN = "979-8-89176-283-1",
    abstract = "Large language models (LLMs) achieve high performance through instruction-tuning, which involves learning various tasks using instruction templates. However, these templates often contain task-specific expressions, which are words that frequently appear in certain contexts but do not always convey the actual meaning of that context, even if they seem closely related to the target task. Biases inherent in such instruction templates may be learned by LLMs during training, potentially degrading performance when the models encounter superficial expressions. In this study, we propose a method that incorporates additional instructions to FLAN templates, without altering the base instruction to produce ``superfluous instructions''. This allows us to investigate the vulnerabilities of LLMs caused by overfitting to task-specific expressions embedded in instruction templates. The experimental results revealed that the inclusion of superficial words strongly related to each task in the instruction text can alter the output, regardless of the intended meaning."
}