@inproceedings{atil-etal-2025-non,
    title = "Non-Determinism of ``Deterministic'' {LLM} System Settings in Hosted Environments",
    author = "At{\i}l, Berk  and
      Aykent, Sarp  and
      Chittams, Alexa  and
      Fu, Lisheng  and
      Passonneau, Rebecca J.  and
      Radcliffe, Evan  and
      Rajagopal, Guru Rajan  and
      Sloan, Adam  and
      Tudrej, Tomasz  and
      Ture, Ferhan  and
      Wu, Zhe  and
      Xu, Lixinyu  and
      Baldwin, Breck",
    editor = "Akter, Mousumi  and
      Chowdhury, Tahiya  and
      Eger, Steffen  and
      Leiter, Christoph  and
      Opitz, Juri  and
      {\c{C}}ano, Erion",
    booktitle = "Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.eval4nlp-1.12/",
    pages = "135--148",
    ISBN = "979-8-89176-305-0",
    abstract = "LLM (large language model) users of hosted providers commonly notice that outputs can vary for the same inputs under settings expected to be deterministic. While it is difficult to get exact statistics, recent reports on specialty news sites and discussion boards suggest that among users in all communities, the majority of LLM usage today is through cloud-based APIs. Yet the questions of how pervasive non- determinism is, and how much it affects perfor- mance results, have not to our knowledge been systematically investigated. We apply five API- based LLMs configured to be deterministic to eight diverse tasks across 10 runs. Experiments reveal accuracy variations of up to 15{\%} across runs, with a gap of up to 70{\%} between best pos- sible performance and worst possible perfor- mance. No LLM consistently delivers the same outputs or accuracies, regardless of task. We speculate about the sources of non-determinism such as input buffer packing across multiple jobs. To better quantify our observations, we introduce metrics focused on quantifying de- terminism, TARr@N for the total agreement rate at N runs over raw output, and TARa@N for total agreement rate of parsed-out answers. Our code and data will be publicly available at https://github.com/Anonymous."
}