Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop

Damian Y. Romero Diaz, Magdalena Anioł, John Culnan


Abstract
We present our experience as annotators in the creation of high-quality, adversarial machine-reading-comprehension data for extractive QA for Task 1 of the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an emergent data collection paradigm with both models and humans in the loop. We set up a quasi-experimental annotation design and perform quantitative analyses across groups with different numbers of annotators focusing on successful adversarial attacks, cost analysis, and annotator confidence correlation. We further perform a qualitative analysis of our perceived difficulty of the task given the different topics of the passages in our dataset and conclude with recommendations and suggestions that might be of value to people working on future DADC tasks and related annotation interfaces.
Anthology ID:
2022.dadc-1.6
Volume:
Proceedings of the First Workshop on Dynamic Adversarial Data Collection
Month:
July
Year:
2022
Address:
Seattle, WA
Venue:
DADC
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
53–60
Language:
URL:
https://aclanthology.org/2022.dadc-1.6
DOI:
10.18653/v1/2022.dadc-1.6
Bibkey:
Cite (ACL):
Damian Y. Romero Diaz, Magdalena Anioł, and John Culnan. 2022. Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop. In Proceedings of the First Workshop on Dynamic Adversarial Data Collection, pages 53–60, Seattle, WA. Association for Computational Linguistics.
Cite (Informal):
Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop (Romero Diaz et al., DADC 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2022.dadc-1.6.pdf