Better Together: Towards Localizing Fact-Related Hallucinations using Open Small Language Models

David Kletz, Sandra Mitrovic, Ljiljana Dolamic, Fabio Rinaldi


Abstract
In this paper, we explore the potential of Open-source Small Language Models (OSLMs) for localizing hallucinations related to factual accuracy. We first present Lucifer, a dataset designed to enable proper and consistent evaluation of LMs, composed of an automatically constructed portion and a manually curated subset intended for qualitative analysis.We then assess the performance of five OSLMs using four carefully designed prompts. Results are evaluated either individually or merged through a voting-based merging approach. While our results demonstrate that the merging method yields promising performance even with smaller models, our manually curated dataset highlights the inherent difficulty of the task, underscoring the need for further research.
Anthology ID:
2025.chomps-main.3
Volume:
Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025)
Month:
December
Year:
2025
Address:
Mumbai, India
Editors:
Aman Sinha, Raúl Vázquez, Timothee Mickus, Rohit Agarwal, Ioana Buhnila, Patrícia Schmidtová, Federica Gamba, Dilip K. Prasad, Jörg Tiedemann
Venues:
CHOMPS | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
20–34
Language:
URL:
https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.chomps-main.3/
DOI:
Bibkey:
Cite (ACL):
David Kletz, Sandra Mitrovic, Ljiljana Dolamic, and Fabio Rinaldi. 2025. Better Together: Towards Localizing Fact-Related Hallucinations using Open Small Language Models. In Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025), pages 20–34, Mumbai, India. Association for Computational Linguistics.
Cite (Informal):
Better Together: Towards Localizing Fact-Related Hallucinations using Open Small Language Models (Kletz et al., CHOMPS 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.chomps-main.3.pdf