Abstract
Social media sites like Twitter possess the potential to complement surveys that measure political opinions and, more specifically, political actors’ approval. However, new challenges related to the reliability and validity of social-media-based estimates arise. Various sentiment analysis and stance detection methods have been developed and used in previous research to measure users’ political opinions based on their content on social media. In this work, we attempt to gauge the efficacy of untargeted sentiment, targeted sentiment, and stance detection methods in labeling various political actors’ approval by benchmarking them across several datasets. We also contrast the performance of these pretrained methods that can be used in an off-the-shelf (OTS) manner against a set of models trained on minimal custom data. We find that OTS methods have low generalizability on unseen and familiar targets, while low-resource custom models are more robust. Our work sheds light on the strengths and limitations of existing methods proposed for understanding politicians’ approval from tweets.- Anthology ID:
- 2020.emnlp-main.110
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1413–1426
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.110
- DOI:
- 10.18653/v1/2020.emnlp-main.110
- Cite (ACL):
- Indira Sen, Fabian Flöck, and Claudia Wagner. 2020. On the Reliability and Validity of Detecting Approval of Political Actors in Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1413–1426, Online. Association for Computational Linguistics.
- Cite (Informal):
- On the Reliability and Validity of Detecting Approval of Political Actors in Tweets (Sen et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2020.emnlp-main.110.pdf