@inproceedings{kapoor-etal-2024-calibration,
    title = "Calibration-Tuning: Teaching Large Language Models to Know What They Don{'}t Know",
    author = "Kapoor, Sanyam  and
      Gruver, Nate  and
      Roberts, Manley  and
      Pal, Arka  and
      Dooley, Samuel  and
      Goldblum, Micah  and
      Wilson, Andrew",
    editor = {V{\'a}zquez, Ra{\'u}l  and
      Celikkanat, Hande  and
      Ulmer, Dennis  and
      Tiedemann, J{\"o}rg  and
      Swayamdipta, Swabha  and
      Aziz, Wilker  and
      Plank, Barbara  and
      Baan, Joris  and
      de Marneffe, Marie-Catherine},
    booktitle = "Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024)",
    month = mar,
    year = "2024",
    address = "St Julians, Malta",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.uncertainlp-1.1",
    pages = "1--14",
    abstract = "Large language models are increasingly deployed for high-stakes decision making, for example in financial and medical applications. In such applications, it is imperative that we be able to estimate our confidence in the answers output by a language model in order to assess risks. Although we can easily compute the probability assigned by a language model to the sequence of tokens that make up an answer, we cannot easily compute the probability of the answer itself, which could be phrased in numerous ways.While other works have engineered ways of assigning such probabilities to LLM outputs, a key problem remains: existing language models are poorly calibrated, often confident when they are wrong or unsure when they are correct. In this work, we devise a protocol called *calibration tuning* for finetuning LLMs to output calibrated probabilities. Calibration-tuned models demonstrate superior calibration performance compared to existing language models on a variety of question-answering tasks, including open-ended generation, without affecting accuracy. We further show that this ability transfers to new domains outside of the calibration-tuning train set.",
}