Jingyi Li
2020
Fact-based Content Weighting for Evaluating Abstractive Summarisation
Xinnuo Xu
|
Ondřej Dušek
|
Jingyi Li
|
Verena Rieser
|
Ioannis Konstas
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Abstractive summarisation is notoriously hard to evaluate since standard word-overlap-based metrics are insufficient. We introduce a new evaluation metric which is based on fact-level content weighting, i.e. relating the facts of the document to the facts of the summary. We fol- low the assumption that a good summary will reflect all relevant facts, i.e. the ones present in the ground truth (human-generated refer- ence summary). We confirm this hypothe- sis by showing that our weightings are highly correlated to human perception and compare favourably to the recent manual highlight- based metric of Hardy et al. (2019).