A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels
Youxuan Jiang, Huaiyu Zhu, Jonathan K. Kummerfeld, Yunyao Li, Walter Lasecki
Abstract
Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great expense. Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. We propose a new multi-stage crowd workflow that substantially reduces expert involvement without sacrificing accuracy. In particular, we introduce a unique filter stage based on the key observation that crowd workers are able to almost perfectly filter out incorrect options for labels. Our three-stage workflow produces annotations with 95% accuracy for predicate labels and 93% for argument labels, which is comparable to expert agreement. Compared to prior work on crowdsourcing for SRL, we decrease expert effort by 4x, from 56% to 14% of cases. Our approach enables more scalable annotation of SRL, and could enable annotation of NLP tasks that have previously been considered too complex to effectively crowdsource.- Anthology ID:
- 2020.findings-emnlp.38
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Trevor Cohn, Yulan He, Yang Liu
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 415–421
- Language:
- URL:
- https://aclanthology.org/2020.findings-emnlp.38
- DOI:
- 10.18653/v1/2020.findings-emnlp.38
- Cite (ACL):
- Youxuan Jiang, Huaiyu Zhu, Jonathan K. Kummerfeld, Yunyao Li, and Walter Lasecki. 2020. A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 415–421, Online. Association for Computational Linguistics.
- Cite (Informal):
- A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels (Jiang et al., Findings 2020)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2020.findings-emnlp.38.pdf
- Code
- system-t/crowdsourcingsrl