Abstract
In this paper we describe our attempt to reproduce a single human evaluation quality criterion of the human evaluation that was in conducted in the paper “NeuralREG: An end-to-end approach to referring expression generation”. In particular, this paper describes the approach and challenges involved in reproducing the human evaluation as done by the original authors of the paper, the results obtained, and what insights we have gained from attempting this particular reproduction. Insights that we hope will enable refinements to both how human evaluations are documented by author(s) and enable better reproductions of NLP experiments in the future.- Anthology ID:
- 2024.humeval-1.22
- Volume:
- Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Simone Balloccu, Anya Belz, Rudali Huidrom, Ehud Reiter, Joao Sedoc, Craig Thomson
- Venues:
- HumEval | WS
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 250–254
- Language:
- URL:
- https://aclanthology.org/2024.humeval-1.22
- DOI:
- Cite (ACL):
- Saad Mahamood. 2024. ReproHum #0124-03: Reproducing Human Evaluations of end-to-end approaches for Referring Expression Generation. In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024, pages 250–254, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- ReproHum #0124-03: Reproducing Human Evaluations of end-to-end approaches for Referring Expression Generation (Mahamood, HumEval-WS 2024)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2024.humeval-1.22.pdf