Abstract
The task of summarisation is notoriously difficult to evaluate, with agreement even between expert raters unlikely to be perfect. One technique for summary evaluation relies on collecting comparison data by presenting annotators with generated summaries and tasking them with selecting the best one. This paradigm is currently being exploited in reinforcement learning using human feedback, whereby a reward function is trained using pairwise choice data. Comparisons are an easier way to elicit human feedback for summarisation, however, such decisions can be bottle necked by the usability of the annotator interface. In this paper, we present the results of a pilot study exploring how the user interface impacts annotator agreement when judging summary quality.- Anthology ID:
- 2023.law-1.18
- Volume:
- Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Jakob Prange, Annemarie Friedrich
- Venue:
- LAW
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 179–187
- Language:
- URL:
- https://aclanthology.org/2023.law-1.18
- DOI:
- 10.18653/v1/2023.law-1.18
- Cite (ACL):
- Sian Gooding, Lucas Werner, and Victor Cărbune. 2023. A Study on Annotation Interfaces for Summary Comparison. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 179–187, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- A Study on Annotation Interfaces for Summary Comparison (Gooding et al., LAW 2023)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2023.law-1.18.pdf