@inproceedings{sadjoli-etal-2025-optimization,
    title = "Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading",
    author = "Sadjoli, Nicholas  and
      Siefken, Tim  and
      Ghosh, Atin  and
      Mai, Yifan  and
      Dahlmeier, Daniel",
    editor = "Rehm, Georg  and
      Li, Yunyao",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-08/2025.acl-industry.44/",
    doi = "10.18653/v1/2025.acl-industry.44",
    pages = "619--638",
    ISBN = "979-8-89176-288-6",
    abstract = "Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task."
}