Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking

Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, Kyomin Jung


Abstract
Despite the recent advances in abstractive summarization systems, it is still difficult to determine whether a generated summary is factual consistent with the source text. To this end, the latest approach is to train a factual consistency classifier on factually consistent and inconsistent summaries. Luckily, the former is readily available as reference summaries in existing summarization datasets. However, generating the latter remains a challenge, as they need to be factually inconsistent, yet closely relevant to the source text to be effective. In this paper, we propose to generate factually inconsistent summaries using source texts and reference summaries with key information masked. Experiments on seven benchmark datasets demonstrate that factual consistency classifiers trained on summaries generated using our method generally outperform existing models and show a competitive correlation with human judgments. We also analyze the characteristics of the summaries generated using our method. We will release the pre-trained model and the code at https://github.com/hwanheelee1993/MFMA.
Anthology ID:
2022.findings-naacl.76
Volume:
Findings of the Association for Computational Linguistics: NAACL 2022
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1019–1030
Language:
URL:
https://aclanthology.org/2022.findings-naacl.76
DOI:
10.18653/v1/2022.findings-naacl.76
Bibkey:
Cite (ACL):
Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. 2022. Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1019–1030, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking (Lee et al., Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/2022.findings-naacl.76.pdf
Software:
 2022.findings-naacl.76.software.tgz
Video:
 https://preview.aclanthology.org/naacl-24-ws-corrections/2022.findings-naacl.76.mp4
Code
 hwanheelee1993/mfma
Data
CNN/Daily MailMultiNLISummEval