longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks.
Venelin Kovatchev, Trina Chatterjee, Venkata S Govindarajan, Jifan Chen, Eunsol Choi, Gabriella Chronis, Anubrata Das, Katrin Erk, Matthew Lease, Junyi Jessy Li, Yating Wu, Kyle Mahowald
Abstract
Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team “longhorns” on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first (pending validation), with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.- Anthology ID:
- 2022.dadc-1.5
- Volume:
- Proceedings of the First Workshop on Dynamic Adversarial Data Collection
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, WA
- Venue:
- DADC
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 41–52
- Language:
- URL:
- https://aclanthology.org/2022.dadc-1.5
- DOI:
- 10.18653/v1/2022.dadc-1.5
- Cite (ACL):
- Venelin Kovatchev, Trina Chatterjee, Venkata S Govindarajan, Jifan Chen, Eunsol Choi, Gabriella Chronis, Anubrata Das, Katrin Erk, Matthew Lease, Junyi Jessy Li, Yating Wu, and Kyle Mahowald. 2022. longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks.. In Proceedings of the First Workshop on Dynamic Adversarial Data Collection, pages 41–52, Seattle, WA. Association for Computational Linguistics.
- Cite (Informal):
- longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. (Kovatchev et al., DADC 2022)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2022.dadc-1.5.pdf