Abstract
This paper reports results from a reproduction study in which we repeated the human evaluation of the PASS Dutch-language football report generation system (van der Lee et al., 2017). The work was carried out as part of the ReproGen Shared Task on Reproducibility of Human Evaluations in NLG, in Track A (Paper 1). We aimed to repeat the original study exactly, with the main difference that a different set of evaluators was used. We describe the study design, present the results from the original and the reproduction study, and then compare and analyse the differences between the two sets of results. For the two ‘headline’ results of average Fluency and Clarity, we find that in both studies, the system was rated more highly for Clarity than for Fluency, and Clarity had higher standard deviation. Clarity and Fluency ratings were higher, and their standard deviations lower, in the reproduction study than in the original study by substantial margins. Clarity had a higher degree of reproducibility than Fluency, as measured by the coefficient of variation. Data and code are publicly available.- Anthology ID:
- 2021.inlg-1.30
- Volume:
- Proceedings of the 14th International Conference on Natural Language Generation
- Month:
- August
- Year:
- 2021
- Address:
- Aberdeen, Scotland, UK
- Editors:
- Anya Belz, Angela Fan, Ehud Reiter, Yaji Sripada
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 286–292
- Language:
- URL:
- https://aclanthology.org/2021.inlg-1.30
- DOI:
- 10.18653/v1/2021.inlg-1.30
- Cite (ACL):
- Simon Mille, Thiago Castro Ferreira, Anya Belz, and Brian Davis. 2021. Another PASS: A Reproduction Study of the Human Evaluation of a Football Report Generation System. In Proceedings of the 14th International Conference on Natural Language Generation, pages 286–292, Aberdeen, Scotland, UK. Association for Computational Linguistics.
- Cite (Informal):
- Another PASS: A Reproduction Study of the Human Evaluation of a Football Report Generation System (Mille et al., INLG 2021)
- PDF:
- https://preview.aclanthology.org/improve-issue-templates/2021.inlg-1.30.pdf