@inproceedings{yaldiz-etal-2026-balancing,
    title = "Balancing Classification and Calibration Performance in Decision-Making {LLM}s via Calibration Aware Reinforcement Learning",
    author = "Yaldiz, Duygu Nur  and
      Spiliopoulou, Evangelia  and
      Qi, Zheng  and
      Varia, Siddharth  and
      Doss, Srikanth  and
      Pappas, Nikolaos",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.610/",
    pages = "12537--12553",
    ISBN = "979-8-89176-395-1",
    abstract = "Large language models (LLMs) are increasingly deployed in decision-making tasks, where not only accuracy but also reliable confidence estimates are essential. Well-calibrated confidence enables downstream systems to decide when to trust a model and when to defer to fallback mechanisms. In this work, we conduct a systematic study of calibration in two widely used fine-tuning paradigms: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). We show that while RLVR improves task performance, it produces extremely overconfident models, whereas SFT yields substantially better calibration, even under distribution shift, though with smaller performance gains. Through targeted experiments, we diagnose RLVR{'}s failure, showing that decision tokens act as extraction steps of the decision in reasoning traces and do not carry confidence information, which prevents reinforcement learning from surfacing calibrated alternatives. Based on this insight, we propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. Our method preserves RLVR{'}s accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points."
}