Abstract
We present an efficient annotation framework for argument quality, a feature difficult to be measured reliably as per previous work. A stochastic transitivity model is combined with an effective sampling strategy to infer high-quality labels with low effort from crowdsourced pairwise judgments. The model’s capabilities are showcased by compiling Webis-ArgQuality-20, an argument quality corpus that comprises scores for rhetorical, logical, dialectical, and overall quality inferred from a total of 41,859 pairwise judgments among 1,271 arguments. With up to 93% cost savings, our approach significantly outperforms existing annotation procedures. Furthermore, novel insight into argument quality is provided through statistical analysis, and a new aggregation method to infer overall quality from individual quality dimensions is proposed.- Anthology ID:
- 2020.acl-main.511
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5772–5781
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.511
- DOI:
- 10.18653/v1/2020.acl-main.511
- Cite (ACL):
- Lukas Gienapp, Benno Stein, Matthias Hagen, and Martin Potthast. 2020. Efficient Pairwise Annotation of Argument Quality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5772–5781, Online. Association for Computational Linguistics.
- Cite (Informal):
- Efficient Pairwise Annotation of Argument Quality (Gienapp et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/starsem-semeval-split/2020.acl-main.511.pdf