Multi-View Source Ablation for Faithful Summarization
Shuyang Cao, Liang Ma, Di Lu, Robert L Logan IV, Joel Tetreault, Alejandro Jaimes
Abstract
In this paper, we present MuFaSSa (Multi-view Faithfulness Scoring via Source Ablation), a metric for evaluating faithfulness of abstractive summaries, and for guiding training of more faithful summarizers. For evaluation, MuFaSSa employs different strategies (e.g., masking entity mentions) to first remove information from the source document to form multiple ablated views. Then, the faithfulness level of each token in a generated summary is measured by the difference between the token generation probabilities when given the original document and the ablated document as inputs to trained summarizers. For training, MuFaSSa uses a novel word truncation objective that drops unfaithful tokens located by MuFaSSa in both the decoder input and output. Alignments with human-annotated faithfulness labels on AggreFact show that MuFaSSa is comparable to or better than existing metrics built on classifiers or QA models pre-trained on other tasks. In experiments on summarization with XSum and CNN/DailyMail, models trained with word truncation using MuFaSSa outperform competitive methods according to both automatic faithfulness metrics and human assessments.- Anthology ID:
- 2023.findings-eacl.151
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2023
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2029–2047
- Language:
- URL:
- https://aclanthology.org/2023.findings-eacl.151
- DOI:
- Cite (ACL):
- Shuyang Cao, Liang Ma, Di Lu, Robert L Logan IV, Joel Tetreault, and Alejandro Jaimes. 2023. Multi-View Source Ablation for Faithful Summarization. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2029–2047, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Multi-View Source Ablation for Faithful Summarization (Cao et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2023.findings-eacl.151.pdf