On the Reliability and Validity of Detecting Approval of Political Actors in Tweets

Indira Sen, Fabian Flöck, Claudia Wagner


Abstract
Social media sites like Twitter possess the potential to complement surveys that measure political opinions and, more specifically, political actors’ approval. However, new challenges related to the reliability and validity of social-media-based estimates arise. Various sentiment analysis and stance detection methods have been developed and used in previous research to measure users’ political opinions based on their content on social media. In this work, we attempt to gauge the efficacy of untargeted sentiment, targeted sentiment, and stance detection methods in labeling various political actors’ approval by benchmarking them across several datasets. We also contrast the performance of these pretrained methods that can be used in an off-the-shelf (OTS) manner against a set of models trained on minimal custom data. We find that OTS methods have low generalizability on unseen and familiar targets, while low-resource custom models are more robust. Our work sheds light on the strengths and limitations of existing methods proposed for understanding politicians’ approval from tweets.
Anthology ID:
2020.emnlp-main.110
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1413–1426
Language:
URL:
https://aclanthology.org/2020.emnlp-main.110
DOI:
10.18653/v1/2020.emnlp-main.110
Bibkey:
Cite (ACL):
Indira Sen, Fabian Flöck, and Claudia Wagner. 2020. On the Reliability and Validity of Detecting Approval of Political Actors in Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1413–1426, Online. Association for Computational Linguistics.
Cite (Informal):
On the Reliability and Validity of Detecting Approval of Political Actors in Tweets (Sen et al., EMNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.emnlp-main.110.pdf
Video:
 https://slideslive.com/38938774