Jay Pavagadhi
2024
Multilingual Fine-Grained News Headline Hallucination Detection
Jiaming Shen
|
Tianqi Liu
|
Jialu Liu
|
Zhen Qin
|
Jay Pavagadhi
|
Simon Baumgartner
|
Michael Bendersky
Findings of the Association for Computational Linguistics: EMNLP 2024
The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the “hallucination” problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand <article, headline> pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset’s challenges and utilities. Second, we test various large language models’ in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection.
Search
Co-authors
- Jialu Liu 1
- Jiaming Shen 1
- Michael Bendersky 1
- Simon Baumgartner 1
- Tianqi Liu 1
- show all...
- Zhen Qin 1