Abstract
We describe in this paper an attempt to reproduce some of the human of evaluation results from the paper “It’s not Rocket Science: Interpreting Figurative Language in Narratives”. In particular, we describe the methodology used to reproduce the chosen human evaluation, the challenges faced, and the results that were gathered. We will also make some recommendations on the learnings obtained from this reproduction attempt and what improvements are needed to enable more robust reproductions of future NLP human evaluations.- Anthology ID:
- 2023.humeval-1.16
- Volume:
- Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems
- Month:
- September
- Year:
- 2023
- Address:
- Varna, Bulgaria
- Editors:
- Anya Belz, Maja Popović, Ehud Reiter, Craig Thomson, João Sedoc
- Venues:
- HumEval | WS
- SIG:
- Publisher:
- INCOMA Ltd., Shoumen, Bulgaria
- Note:
- Pages:
- 204–209
- Language:
- URL:
- https://aclanthology.org/2023.humeval-1.16
- DOI:
- Cite (ACL):
- Saad Mahamood. 2023. Reproduction of Human Evaluations in: “It’s not Rocket Science: Interpreting Figurative Language in Narratives”. In Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, pages 204–209, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
- Cite (Informal):
- Reproduction of Human Evaluations in: “It’s not Rocket Science: Interpreting Figurative Language in Narratives” (Mahamood, HumEval-WS 2023)
- PDF:
- https://preview.aclanthology.org/ingest-bitext-workshop/2023.humeval-1.16.pdf