A Streamlined Method for Sourcing Discourse-level Argumentation Annotations from the Crowd

Tristan Miller, Maria Sukhareva, Iryna Gurevych

Abstract
The study of argumentation and the development of argument mining tools depends on the availability of annotated data, which is challenging to obtain in sufficient quantity and quality. We present a method that breaks down a popular but relatively complex discourse-level argument annotation scheme into a simpler, iterative procedure that can be applied even by untrained annotators. We apply this method in a crowdsourcing setup and report on the reliability of the annotations obtained. The source code for a tool implementing our annotation method, as well as the sample data we obtained (4909 gold-standard annotations across 982 documents), are freely released to the research community. These are intended to serve the needs of qualitative research into argumentation, as well as of data-driven approaches to argument mining.
Anthology ID:
N19-1177
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Jill Burstein, Christy Doran, Thamar Solorio
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1790–1796
Language:
URL:
https://aclanthology.org/N19-1177
DOI:
10.18653/v1/N19-1177
Bibkey:
Cite (ACL):
Tristan Miller, Maria Sukhareva, and Iryna Gurevych. 2019. A Streamlined Method for Sourcing Discourse-level Argumentation Annotations from the Crowd. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1790–1796, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
A Streamlined Method for Sourcing Discourse-level Argumentation Annotations from the Crowd (Miller et al., NAACL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-3/N19-1177.pdf
Code
 UKPLab/naacl2019-argument-annotations
PDF Search Code