A Parameterized and Annotated Spoken Dialog Corpus of the CMU Let’s Go Bus Information System

Alexander Schmitt, Stefan Ultes, Wolfgang Minker

Abstract
Standardized corpora are the foundation for spoken language research. In this work, we introduce an annotated and standardized corpus in the Spoken Dialog Systems (SDS) domain. Data from the Let's Go Bus Information System from the Carnegie Mellon University in Pittsburgh has been formatted, parameterized and annotated with quality, emotion, and task success labels containing 347 dialogs with 9,083 system-user exchanges. A total of 46 parameters have been derived automatically and semi-automatically from Automatic Speech Recognition (ASR), Spoken Language Understanding (SLU) and Dialog Manager (DM) properties. To each spoken user utterance an emotion label from the set garbage, non-angry, slightly angry, very angry has been assigned. In addition, a manual annotation of Interaction Quality (IQ) on the exchange level has been performed with three raters achieving a Kappa value of 0.54. The IQ score expresses the quality of the interaction up to each system-user exchange on a score from 1-5. The presented corpus is intended as a standardized basis for classification and evaluation tasks regarding task success prediction, dialog quality estimation or emotion recognition to foster comparability between different approaches on these fields.
Anthology ID:
L12-1157
Volume:
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
Month:
May
Year:
2012
Address:
Istanbul, Turkey
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
3369–3373
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/333_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Alexander Schmitt, Stefan Ultes, and Wolfgang Minker. 2012. A Parameterized and Annotated Spoken Dialog Corpus of the CMU Let’s Go Bus Information System. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3369–3373, Istanbul, Turkey. European Language Resources Association (ELRA).
Cite (Informal):
A Parameterized and Annotated Spoken Dialog Corpus of the CMU Let’s Go Bus Information System (Schmitt et al., LREC 2012)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/333_Paper.pdf
PDF Search