Evaluating Factuality in Cross-lingual Summarization

Mingqi Gao, Wenqing Wang, Xiaojun Wan, Yuemei Xu

Abstract
Cross-lingual summarization aims to help people efficiently grasp the core idea of the document written in a foreign language. Modern text summarization models generate highly fluent but often factually inconsistent outputs, which has received heightened attention in recent research. However, the factual consistency of cross-lingual summarization has not been investigated yet. In this paper, we propose a cross-lingual factuality dataset by collecting human annotations of reference summaries as well as generated summaries from models at both summary level and sentence level. Furthermore, we perform the fine-grained analysis and observe that over 50% of generated summaries and over 27% of reference summaries contain factual errors with characteristics different from monolingual summarization. Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of cross-lingual summarization and perform differently at different tasks and levels. Finally, we adapt the monolingual factuality metrics as an initial step towards the automatic evaluation of summarization factuality in cross-lingual settings. Our dataset and code are available at https://github.com/kite99520/Fact_CLS.
Anthology ID:
2023.findings-acl.786
Volume:
Findings of the Association for Computational Linguistics: ACL 2023
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12415–12431
Language:
URL:
https://aclanthology.org/2023.findings-acl.786
DOI:
10.18653/v1/2023.findings-acl.786
Bibkey:
Cite (ACL):
Mingqi Gao, Wenqing Wang, Xiaojun Wan, and Yuemei Xu. 2023. Evaluating Factuality in Cross-lingual Summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12415–12431, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
Evaluating Factuality in Cross-lingual Summarization (Gao et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/improve-issue-templates/2023.findings-acl.786.pdf
PDF Search