@inproceedings{alyahya-etal-2025-zerosumeval,
    title = "{Z}ero{S}um{E}val: An Extensible Framework For Scaling {LLM} Evaluation with Inter-Model Competition",
    author = "Alyahya, Hisham Abdullah  and
      Khan, Haidar  and
      Alnumay, Yazeed  and
      Bari, M Saiful  and
      Yener, Bulent",
    editor = "Mishra, Pushkar  and
      Muresan, Smaranda  and
      Yu, Tao",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-demo.33/",
    pages = "340--350",
    ISBN = "979-8-89176-253-4",
    abstract = "We introduce ZeroSumEval, a dynamic, competition-based, and evolving evaluation framework for Large Language Models (LLMs) that leverages competitive games. ZeroSumEval encompasses a diverse suite of games, including security challenges (Capture the Flag), classic board games (chess), and knowledge tests (MathQuiz). These games are designed to evaluate a range of capabilities such as strategic reasoning, planning, knowledge application, safety, and adaptability. Building upon recent studies that highlight the effectiveness of game-based evaluations for LLMs, ZeroSumEval enhances these approaches by providing a standardized and extensible framework for easily implementing games and leverages DSPy to provide a better abstraction for LLM player strategies."
}