Better Together: Towards Localizing Fact-Related Hallucinations using Open Small Language Models
David Kletz, Sandra Mitrovic, Ljiljana Dolamic, Fabio Rinaldi
Abstract
In this paper, we explore the potential of Open-source Small Language Models (OSLMs) for localizing hallucinations related to factual accuracy. We first present Lucifer, a dataset designed to enable proper and consistent evaluation of LMs, composed of an automatically constructed portion and a manually curated subset intended for qualitative analysis.We then assess the performance of five OSLMs using four carefully designed prompts. Results are evaluated either individually or merged through a voting-based merging approach. While our results demonstrate that the merging method yields promising performance even with smaller models, our manually curated dataset highlights the inherent difficulty of the task, underscoring the need for further research.- Anthology ID:
- 2025.chomps-main.3
- Volume:
- Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025)
- Month:
- December
- Year:
- 2025
- Address:
- Mumbai, India
- Editors:
- Aman Sinha, Raúl Vázquez, Timothee Mickus, Rohit Agarwal, Ioana Buhnila, Patrícia Schmidtová, Federica Gamba, Dilip K. Prasad, Jörg Tiedemann
- Venues:
- CHOMPS | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 20–34
- Language:
- URL:
- https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.chomps-main.3/
- DOI:
- Cite (ACL):
- David Kletz, Sandra Mitrovic, Ljiljana Dolamic, and Fabio Rinaldi. 2025. Better Together: Towards Localizing Fact-Related Hallucinations using Open Small Language Models. In Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025), pages 20–34, Mumbai, India. Association for Computational Linguistics.
- Cite (Informal):
- Better Together: Towards Localizing Fact-Related Hallucinations using Open Small Language Models (Kletz et al., CHOMPS 2025)
- PDF:
- https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.chomps-main.3.pdf