@inproceedings{lovering-etal-2025-language,
    title = "Language Model Probabilities are $Not$ Calibrated in Numeric Contexts",
    author = "Lovering, Charles  and
      Krumdick, Michael  and
      Lai, Viet Dac  and
      Reddy, Varshini  and
      Ebner, Seth  and
      Kumar, Nilesh  and
      Koncel-Kedziorski, Rik  and
      Tanner, Chris",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1417/",
    pages = "29218--29257",
    ISBN = "979-8-89176-251-0",
    abstract = "Some statements have one well-defined continuation (e.g., ``the Eiffel Tower is in [$Paris$]''), whereas others have a natural distribution over multiple options (e.g., ``the weighted coin flip was [$Heads/Tails$].'') We argue that language model (LM) outputs should capture these natural distributions. Our work specifically tests whether LM output probabilities are calibrated to numeric information within their textual contexts. For example, if the context (the prompt) concerns two equally likely options (e.g., heads or tails for a fair coin), the LM output probabilities should also be equal. Likewise, in a context with nonuniformly likely events (e.g., rolling a pair with two dice) an LM should output proportionate probabilities. However, we find that even in simple settings, the best LMs (1) are poorly calibrated and (2) have systematic biases: artifacts like word identity, word order, and word frequency all impact calibration. For example, `gpt-4o-mini{`} often picks the first of two options presented in the prompt regardless of the options' implied likelihoods, whereas `Llama-3.1-8B{`} picks the second. Models do not allocate probability mass among valid options in a calibrated manner."
}