@inproceedings{shelmanov-etal-2025-head,
    title = "A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in {LLM} Outputs",
    author = "Shelmanov, Artem  and
      Fadeeva, Ekaterina  and
      Tsvigun, Akim  and
      Tsvigun, Ivan  and
      Xie, Zhuohan  and
      Kiselev, Igor  and
      Daheim, Nico  and
      Zhang, Caiqi  and
      Vazhentsev, Artem  and
      Sachan, Mrinmaya  and
      Nakov, Preslav  and
      Baldwin, Timothy",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1809/",
    pages = "35700--35719",
    ISBN = "979-8-89176-332-6",
    abstract = "LLMs have the tendency to hallucinate, i.e., to sporadically generate false or fabricated information, and users generally lack the tools to detect when this happens. Uncertainty quantification (UQ) provides a framework for assessing the reliability of model outputs, aiding in the identification of potential hallucinations. In this work, we introduce pre-trained UQ heads: supervised auxiliary modules for LLMs that substantially enhance their ability to capture uncertainty compared to unsupervised UQ methods. Their strong performance stems from the transformer architecture in their design, in the form of informative features derived from LLM attention maps and logits. Our experiments show that these heads are highly robust and achieve state-of-the-art performance in claim-level hallucination detection across both in-domain and out-of-domain prompts. Moreover, these modules demonstrate strong generalization to languages they were not explicitly trained on. We pre-train a collection of UQ heads for popular LLM series, including Mistral, Llama, and Gemma. We publicly release both the code and the pre-trained heads."
}