@inproceedings{cecere-etal-2025-monte,
    title = "{M}onte {C}arlo Temperature: a robust sampling strategy for {LLM}{'}s uncertainty quantification methods",
    author = "Cecere, Nicola  and
      Bacciu, Andrea  and
      Fern{\'a}ndez-Tob{\'i}as, Ignacio  and
      Mantrach, Amin",
    editor = "Cao, Trista  and
      Das, Anubrata  and
      Kumarage, Tharindu  and
      Wan, Yixin  and
      Krishna, Satyapriya  and
      Mehrabi, Ninareh  and
      Dhamala, Jwala  and
      Ramakrishna, Anil  and
      Galystan, Aram  and
      Kumar, Anoop  and
      Gupta, Rahul  and
      Chang, Kai-Wei",
    booktitle = "Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025)",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.trustnlp-main.21/",
    pages = "305--320",
    ISBN = "979-8-89176-233-6",
    abstract = "Uncertainty quantification (UQ) in Large Language Models (LLMs) is essential for their safe and reliable deployment, particularly in critical applications where incorrect outputs can have serious consequences. Current UQ methods typically rely on querying the model multiple times using non-zero temperature sampling to generate diverse outputs for uncertainty estimation. However, the impact of selecting a given temperature parameter is understudied, and our analysis reveals that temperature plays a fundamental role in the quality of uncertainty estimates. The conventional approach of identifying optimal temperature values requires expensive hyperparameter optimization (HPO) that must be repeated for each new model-dataset combination. We propose Monte Carlo Temperature (MCT), a robust sampling strategy that eliminates the need for temperature calibration. Our analysis reveals that: 1) MCT provides more robust uncertainty estimates across a wide range of temperatures, 2) MCT improves the performance of UQ methods by replacing fixed-temperature strategies that do not rely on HPO, and 3) MCT achieves statistical parity with oracle temperatures, which represent the ideal outcome of a well-tuned but computationally expensive HPO process. These findings demonstrate that effective UQ can be achieved without the computational burden of temperature parameter calibration."
}