Agree or Disagree: Predicting Judgments on Nuanced Assertions
Michael Wojatzki, Torsten Zesch, Saif Mohammad, Svetlana Kiritchenko
Abstract
Being able to predict whether people agree or disagree with an assertion (i.e. an explicit, self-contained statement) has several applications ranging from predicting how many people will like or dislike a social media post to classifying posts based on whether they are in accordance with a particular point of view. We formalize this as two NLP tasks: predicting judgments of (i) individuals and (ii) groups based on the text of the assertion and previous judgments. We evaluate a wide range of approaches on a crowdsourced data set containing over 100,000 judgments on over 2,000 assertions. We find that predicting individual judgments is a hard task with our best results only slightly exceeding a majority baseline, but that judgments of groups can be more reliably predicted using a Siamese neural network, which outperforms all other approaches by a wide margin.- Anthology ID:
- S18-2026
- Volume:
- Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics
- Month:
- June
- Year:
- 2018
- Address:
- New Orleans, Louisiana
- Editors:
- Malvina Nissim, Jonathan Berant, Alessandro Lenci
- Venue:
- *SEM
- SIGs:
- SIGLEX | SIGSEM
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 214–224
- Language:
- URL:
- https://aclanthology.org/S18-2026
- DOI:
- 10.18653/v1/S18-2026
- Cite (ACL):
- Michael Wojatzki, Torsten Zesch, Saif Mohammad, and Svetlana Kiritchenko. 2018. Agree or Disagree: Predicting Judgments on Nuanced Assertions. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 214–224, New Orleans, Louisiana. Association for Computational Linguistics.
- Cite (Informal):
- Agree or Disagree: Predicting Judgments on Nuanced Assertions (Wojatzki et al., *SEM 2018)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/S18-2026.pdf
- Code
- muchafel/judgmentPrediction