Abstract
This paper presents the Human Evaluation Datasheet (HEDS), a template for recording the details of individual human evaluation experiments in Natural Language Processing (NLP), and reports on first experience of researchers using HEDS sheets in practice. Originally taking inspiration from seminal papers by Bender and Friedman (2018), Mitchell et al. (2019), and Gebru et al. (2020), HEDS facilitates the recording of properties of human evaluations in sufficient detail, and with sufficient standardisation, to support comparability, meta-evaluation,and reproducibility assessments for human evaluations. These are crucial for scientifically principled evaluation, but the overhead of completing a detailed datasheet is substantial, and we discuss possible ways of addressing this and other issues observed in practice.- Anthology ID:
- 2022.humeval-1.6
- Volume:
- Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Anya Belz, Maja Popović, Ehud Reiter, Anastasia Shimorina
- Venue:
- HumEval
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 54–75
- Language:
- URL:
- https://aclanthology.org/2022.humeval-1.6
- DOI:
- 10.18653/v1/2022.humeval-1.6
- Cite (ACL):
- Anastasia Shimorina and Anya Belz. 2022. The Human Evaluation Datasheet: A Template for Recording Details of Human Evaluation Experiments in NLP. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 54–75, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- The Human Evaluation Datasheet: A Template for Recording Details of Human Evaluation Experiments in NLP (Shimorina & Belz, HumEval 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.humeval-1.6.pdf
- Code
- Shimorina/human-evaluation-datasheet