Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale

Gabriella Skitalinskaya, Jonas Klaff, Henning Wachsmuth


Abstract
Assessing the quality of arguments and of the claims the arguments are composed of has become a key task in computational argumentation. However, even if different claims share the same stance on the same topic, their assessment depends on the prior perception and weighting of the different aspects of the topic being discussed. This renders it difficult to learn topic-independent quality indicators. In this paper, we study claim quality assessment irrespective of discussed aspects by comparing different revisions of the same claim. We compile a large-scale corpus with over 377k claim revision pairs of various types from kialo.com, covering diverse topics from politics, ethics, entertainment, and others. We then propose two tasks: (a) assessing which claim of a revision pair is better, and (b) ranking all versions of a claim by quality. Our first experiments with embedding-based logistic regression and transformer-based neural networks show promising results, suggesting that learned indicators generalize well across topics. In a detailed error analysis, we give insights into what quality dimensions of claims can be assessed reliably. We provide the data and scripts needed to reproduce all results.
Anthology ID:
2021.eacl-main.147
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1718–1729
Language:
URL:
https://aclanthology.org/2021.eacl-main.147
DOI:
10.18653/v1/2021.eacl-main.147
Bibkey:
Cite (ACL):
Gabriella Skitalinskaya, Jonas Klaff, and Henning Wachsmuth. 2021. Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1718–1729, Online. Association for Computational Linguistics.
Cite (Informal):
Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale (Skitalinskaya et al., EACL 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/auto-file-uploads/2021.eacl-main.147.pdf
Dataset:
 2021.eacl-main.147.Dataset.zip
Code
 GabriellaSky/claimrev