Abstract
In recent years, researchers have developed question-answering based approaches to automatically evaluate system summaries, reporting improved validity compared to word overlap-based metrics like ROUGE, in terms of correlation with human ratings of criteria including fluency and hallucination. In this paper, we take a closer look at one particular metric, QuestEval, and ask whether: (1) it can serve as a more general metric for long document similarity assessment; and (2) a single correlation score between metric scores and human ratings, as the currently standard approach, is sufficient for metric validation. We find that correlation scores can be misleading, and that score distributions and outliers should be taken into account. With these caveats in mind, QuestEval can be a promising candidate for long document similarity assessment.- Anthology ID:
- 2022.alta-1.18
- Volume:
- Proceedings of the 20th Annual Workshop of the Australasian Language Technology Association
- Month:
- December
- Year:
- 2022
- Address:
- Adelaide, Australia
- Editors:
- Pradeesh Parameswaran, Jennifer Biggs, David Powers
- Venue:
- ALTA
- SIG:
- Publisher:
- Australasian Language Technology Association
- Note:
- Pages:
- 130–138
- Language:
- URL:
- https://aclanthology.org/2022.alta-1.18
- DOI:
- Cite (ACL):
- Gisela Vallejo, Timothy Baldwin, and Lea Frermann. 2022. Evaluating the Examiner: The Perils of Pearson Correlation for Validating Text Similarity Metrics. In Proceedings of the 20th Annual Workshop of the Australasian Language Technology Association, pages 130–138, Adelaide, Australia. Australasian Language Technology Association.
- Cite (Informal):
- Evaluating the Examiner: The Perils of Pearson Correlation for Validating Text Similarity Metrics (Vallejo et al., ALTA 2022)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2022.alta-1.18.pdf