Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection
John Culnan, Seongjin Park, Meghavarshini Krishnaswamy, Rebecca Sharp
Abstract
In deployment, systems that use speech as input must make use of automated transcriptions. Yet, typically when these systems are evaluated, gold transcriptions are assumed. We explicitly examine the impact of transcription errors on the downstream performance of a multi-modal system on three related tasks from three datasets: emotion, sarcasm, and personality detection. We include three separate transcription tools and show that while all automated transcriptions propagate errors that substantially impact downstream performance, the open-source tools fair worse than the paid tool, though not always straightforwardly, and word error rates do not correlate well with downstream performance. We further find that the inclusion of audio features partially mitigates transcription errors, but that a naive usage of a multi-task setup does not.- Anthology ID:
- 2021.wassa-1.26
- Volume:
- Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis
- Month:
- April
- Year:
- 2021
- Address:
- Online
- Editors:
- Orphee De Clercq, Alexandra Balahur, Joao Sedoc, Valentin Barriere, Shabnam Tafreshi, Sven Buechel, Veronique Hoste
- Venue:
- WASSA
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 250–256
- Language:
- URL:
- https://aclanthology.org/2021.wassa-1.26
- DOI:
- Cite (ACL):
- John Culnan, Seongjin Park, Meghavarshini Krishnaswamy, and Rebecca Sharp. 2021. Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 250–256, Online. Association for Computational Linguistics.
- Cite (Informal):
- Me, myself, and ire: Effects of automatic transcription quality on emotion, sarcasm, and personality detection (Culnan et al., WASSA 2021)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2021.wassa-1.26.pdf
- Data
- LibriSpeech, MELD