@inproceedings{arana-etal-2026-models,
    title = "When Models Hesitate: Answer Instability as a Label-Free Uncertainty Signal for {LLM}s",
    author = "Arana, Jasper Meynard  and
      Carandang, Kristine Ann M.  and
      Casin, Ethan Robert  and
      Alis, Christian  and
      Monterola, Christopher",
    editor = "T.Y.S.S., Santosh  and
      Rodriguez, Juan Diego  and
      de Gibert, Ona",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics ({ACL} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-srw.72/",
    pages = "816--826",
    ISBN = "979-8-89176-393-7",
    abstract = "Large language models (LLMs) are increasingly deployed in high-stakes settings, yet reliably estimating when their outputs should be trusted remains an open challenge. Existing uncertainty estimation approaches{---}such as calibration, token-level probabilities, or semantic entropy{---}typically require access to model internals, additional supervision, or computationally intensive pipelines. We propose answer instability, defined as the variability of a model{'}s final answer across repeated stochastic generations of the same prompt, as a simple, label-free, and black-box uncertainty signal. Evaluated across three task types {---} reasoning, multiple-choice QA, and constraint-following {---} using four LLMs and 520 prompt-model pairs, our approach achieves performance competitive with semantic entropy while requiring no semantic similarity model. Our results show that instability strongly correlates with prediction errors and reliably discriminates correct from incorrect outputs. We further demonstrate its utility for selective prediction and targeted repair, improving reliability without access to internal probabilities or additional training."
}