@inproceedings{sadr-etal-2026-zip,
    title = "{ZIP}: Quantifying Which Words Matter in Zero-Shot Instructional Prompts",
    author = "Sadr, Nikta Gohari  and
      Madhusudan, Sangmitra  and
      Asgari, Arash  and
      Sajjad, Hassan  and
      Seyyed-Kalantari, Laleh  and
      Emami, Ali",
    editor = "Mohammad, Saif M.  and
      Ousidhoum, Nedjma",
    booktitle = "Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*{SEM} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.starsem-conference.30/",
    pages = "428--453",
    ISBN = "979-8-89176-413-2",
    abstract = "While zero-shot instructional prompts like ``Let{'}s think step-by-step'' have revolutionized Large Language Model performance, we lack systematic understanding of why: which specific words drive their effectiveness, and how do these patterns vary across tasks and models? We introduce the ZIP score (Zero-shot Importance of Perturbation), a metric that quantifies individual word importance through controlled, semantically meaningful perturbations. To enable rigorous evaluation, we also introduce the first ground-truth benchmark for prompt interpretability, a set of validation prompts with predetermined keywords where ZIP achieves 95.8{\%} accuracy compared to 65.8{\%} for LIME. Analyzing six flagship models across seven prompts and multiple task domains, we find that word importance is task-dependent ({''}step-by-step'' dominates mathematical reasoning; ``think'' matters more for common-sense tasks), varies systematically across model families, and correlates inversely with model performance, suggesting prompts have greatest impact on tasks where models struggle. Our findings advance prompt science, providing both practical guidance for prompt engineering and theoretical understanding of how instructional language shapes model behavior."
}