Abstract
We present our experience as annotators in the creation of high-quality, adversarial machine-reading-comprehension data for extractive QA for Task 1 of the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an emergent data collection paradigm with both models and humans in the loop. We set up a quasi-experimental annotation design and perform quantitative analyses across groups with different numbers of annotators focusing on successful adversarial attacks, cost analysis, and annotator confidence correlation. We further perform a qualitative analysis of our perceived difficulty of the task given the different topics of the passages in our dataset and conclude with recommendations and suggestions that might be of value to people working on future DADC tasks and related annotation interfaces.- Anthology ID:
- 2022.dadc-1.6
- Volume:
- Proceedings of the First Workshop on Dynamic Adversarial Data Collection
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, WA
- Editors:
- Max Bartolo, Hannah Kirk, Pedro Rodriguez, Katerina Margatina, Tristan Thrush, Robin Jia, Pontus Stenetorp, Adina Williams, Douwe Kiela
- Venue:
- DADC
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 53–60
- Language:
- URL:
- https://aclanthology.org/2022.dadc-1.6
- DOI:
- 10.18653/v1/2022.dadc-1.6
- Cite (ACL):
- Damian Y. Romero Diaz, Magdalena Anioł, and John Culnan. 2022. Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop. In Proceedings of the First Workshop on Dynamic Adversarial Data Collection, pages 53–60, Seattle, WA. Association for Computational Linguistics.
- Cite (Informal):
- Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop (Romero Diaz et al., DADC 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2022.dadc-1.6.pdf