Abstract
Neural methods for sentiment analysis have led to quantitative improvements over previous approaches, but these advances are not always accompanied with a thorough analysis of the qualitative differences. Therefore, it is not clear what outstanding conceptual challenges for sentiment analysis remain. In this work, we attempt to discover what challenges still prove a problem for sentiment classifiers for English and to provide a challenging dataset. We collect the subset of sentences that an (oracle) ensemble of state-of-the-art sentiment classifiers misclassify and then annotate them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, modality, etc. Finally, we provide a case study that demonstrates the usefulness of the dataset to probe the performance of a given sentiment classifier with respect to linguistic phenomena.- Anthology ID:
- W19-4802
- Volume:
- Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
- Month:
- August
- Year:
- 2019
- Address:
- Florence, Italy
- Venue:
- BlackboxNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 12–23
- Language:
- URL:
- https://aclanthology.org/W19-4802
- DOI:
- 10.18653/v1/W19-4802
- Cite (ACL):
- Jeremy Barnes, Lilja Øvrelid, and Erik Velldal. 2019. Sentiment Analysis Is Not Solved! Assessing and Probing Sentiment Classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 12–23, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Sentiment Analysis Is Not Solved! Assessing and Probing Sentiment Classification (Barnes et al., BlackboxNLP 2019)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/W19-4802.pdf
- Code
- ltgoslo/assessing_and_probing_sentiment
- Data
- SST