Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles

Carel van Niekerk, Michael Heck, Christian Geishauser, Hsien-chin Lin, Nurul Lubis, Marco Moresi, Milica Gasic

Abstract
The ability to accurately track what happens during a conversation is essential for the performance of a dialogue system. Current state-of-the-art multi-domain dialogue state trackers achieve just over 55% accuracy on the current go-to benchmark, which means that in almost every second dialogue turn they place full confidence in an incorrect dialogue state. Belief trackers, on the other hand, maintain a distribution over possible dialogue states. However, they lack in performance compared to dialogue state trackers, and do not produce well calibrated distributions. In this work we present state-of-the-art performance in calibration for multi-domain dialogue belief trackers using a calibrated ensemble of models. Our resulting dialogue belief tracker also outperforms previous dialogue belief tracking models in terms of accuracy.
Anthology ID:
2020.findings-emnlp.277
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3096–3102
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.277
DOI:
10.18653/v1/2020.findings-emnlp.277
Bibkey:
Cite (ACL):
Carel van Niekerk, Michael Heck, Christian Geishauser, Hsien-chin Lin, Nurul Lubis, Marco Moresi, and Milica Gasic. 2020. Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3096–3102, Online. Association for Computational Linguistics.
Cite (Informal):
Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles (van Niekerk et al., Findings 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/remove-xml-comments/2020.findings-emnlp.277.pdf
PDF Search