Ruixuan Tu
2023
DocAsRef: An Empirical Study on Repurposing Reference-based Summary Quality Metrics as Reference-free Metrics
Forrest Bao
|
Ruixuan Tu
|
Ge Luo
|
Yinfei Yang
|
Hebi Li
|
Minghui Qiu
|
Youbiao He
|
Cen Chen
Findings of the Association for Computational Linguistics: EMNLP 2023
Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of <0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.
Search
Co-authors
- Forrest Bao 1
- Ge Luo 1
- Yinfei Yang 1
- Hebi Li 1
- Minghui Qiu 1
- show all...