Eliciting and Annotating Uncertainty in Spoken Language

Heather Pon-Barry, Stuart Shieber, Nicholas Longenbaugh

Abstract
A major challenge in the field of automatic recognition of emotion and affect in speech is the subjective nature of affect labels. The most common approach to acquiring affect labels is to ask a panel of listeners to rate a corpus of spoken utterances along one or more dimensions of interest. For applications ranging from educational technology to voice search to dictation, a speaker’s level of certainty is a primary dimension of interest. In such applications, we would like to know the speaker’s actual level of certainty, but past research has only revealed listeners’ perception of the speaker’s level of certainty. In this paper, we present a method for eliciting spoken utterances using stimuli that we design such that they have a quantitative, crowdsourced legibility score. While we cannot control a speaker’s actual internal level of certainty, the use of these stimuli provides a better estimate of internal certainty compared to existing speech corpora. The Harvard Uncertainty Speech Corpus, containing speech data, certainty annotations, and prosodic features, is made available to the research community.
Anthology ID:
L14-1118
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
1978–1983
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1167_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Heather Pon-Barry, Stuart Shieber, and Nicholas Longenbaugh. 2014. Eliciting and Annotating Uncertainty in Spoken Language. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1978–1983, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
Eliciting and Annotating Uncertainty in Spoken Language (Pon-Barry et al., LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1167_Paper.pdf
PDF Search