@inproceedings{tan-etal-2026-basecal,
    title = "{B}ase{C}al: Unsupervised Confidence Calibration via Base Model Signals",
    author = "Tan, Hexiang  and
      Yang, Wanli  and
      Zhang, Junwei  and
      Chen, Xin  and
      Tang, Rui  and
      Su, Du  and
      Wang, Jingang  and
      Wang, Yuanzhuo  and
      Sun, Fei  and
      Cheng, Xueqi",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.234/",
    pages = "5172--5187",
    ISBN = "979-8-89176-390-6",
    abstract = "Reliable confidence is essential for trusting the outputs of LLMs, yet widely deployed post-trained LLMs (PoLLMs) typically compromise this trust with severe overconfidence. In contrast, we observe that their corresponding base LLMs often remain well-calibrated. This naturally motivates us to calibrate PoLLM confidence using the base LLM as a reference. This work proposes two ways to achieve this. A straightforward solution, BaseCal-ReEval, evaluates PoLLM{'}s responses by feeding them into the base LLM to get average probabilities as confidence. While effective, this approach introduces additional inference overhead. To address this, we propose BaseCal-Proj, which trains a lightweight projection to map the final-layer hidden states of PoLLMs back to those of their base LLMs. These projected states are then processed by the base LLM{'}s output layer to derive base-calibrated confidence for PoLLM{'}s responses. Notably, BaseCal is an unsupervised, plug-and-play solution that operates without human labels or LLM modifications. Experiments across five datasets and three LLM families demonstrate the effectiveness of BaseCal, reducing Expected Calibration Error (ECE) by an average of 42.90{\%} compared to the best unsupervised baselines."
}