DSTC-11: Speech Aware Task-Oriented Dialog Modeling Track
Hagen Soltau, Izhak Shafran, Mingqiu Wang, Abhinav Rastogi, Wei Han, Yuan Cao
Abstract
Most research on task oriented dialog modeling is based on written text input. However, users interact with practical dialog systems often using speech as input. Typically, systems convert speech into text using an Automatic Speech Recognition (ASR) system, introducing errors. Furthermore, these systems do not address the differences in written and spoken language. The research on this topic is stymied by the lack of a public corpus. Motivated by these considerations, our goal in hosting the speech-aware dialog state tracking challenge was to create a public corpus or task which can be used to investigate the performance gap between the written and spoken forms of input, develop models that could alleviate this gap, and establish whether Text-to-Speech-based (TTS) systems is a reasonable surrogate to the more-labor intensive human data collection. We created three spoken versions of the popular written-domain MultiWoz task – (a) TTS-Verbatim: written user inputs were converted into speech waveforms using a TTS system, (b) Human-Verbatim: humans spoke the user inputs verbatim, and (c) Human-paraphrased: humans paraphrased the user inputs. Additionally, we provided different forms of ASR output to encourage wider participation from teams that may not have access to state-of-the-art ASR systems. These included ASR transcripts, word time stamps, and latent representations of the audio (audio encoder outputs). In this paper, we describe the corpus, report results from participating teams, provide preliminary analyses of their results, and summarize the current state-of-the-art in this domain.- Anthology ID:
- 2023.dstc-1.25
- Volume:
- Proceedings of The Eleventh Dialog System Technology Challenge
- Month:
- September
- Year:
- 2023
- Address:
- Prague, Czech Republic
- Editors:
- Yun-Nung Chen, Paul Crook, Michel Galley, Sarik Ghazarian, Chulaka Gunasekara, Raghav Gupta, Behnam Hedayatnia, Satwik Kottur, Seungwhan Moon, Chen Zhang
- Venues:
- DSTC | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 226–234
- Language:
- URL:
- https://aclanthology.org/2023.dstc-1.25
- DOI:
- Cite (ACL):
- Hagen Soltau, Izhak Shafran, Mingqiu Wang, Abhinav Rastogi, Wei Han, and Yuan Cao. 2023. DSTC-11: Speech Aware Task-Oriented Dialog Modeling Track. In Proceedings of The Eleventh Dialog System Technology Challenge, pages 226–234, Prague, Czech Republic. Association for Computational Linguistics.
- Cite (Informal):
- DSTC-11: Speech Aware Task-Oriented Dialog Modeling Track (Soltau et al., DSTC-WS 2023)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2023.dstc-1.25.pdf