WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue

Anant Khandelwal


Abstract
An intelligent dialogue system in a multi-turn setting should not only generate the responses which are of good quality, but it should also generate the responses which can lead to long-term success of the dialogue. Although, the current approaches improved the response quality, but they over-look the training signals present in the dialogue data. We can leverage these signals to generate the weakly supervised training data for learning dialog policy and reward estimator, and make the policy take actions (generates responses) which can foresee the future direction for a successful (rewarding) conversation. We simulate the dialogue between an agent and a user (modelled similar to an agent with supervised learning objective) to interact with each other. The agent uses dynamic blocking to generate ranked diverse responses and exploration-exploitation to select among the Top-K responses. Each simulated state-action pair is evaluated (works as a weak annotation) with three quality modules: Semantic Relevant, Semantic Coherence and Consistent Flow. Empirical studies with two benchmarks indicate that our model can significantly out-perform the response quality and lead to a successful conversation on both automatic evaluation and human judgment.
Anthology ID:
2021.dialdoc-1.10
Volume:
Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP | dialdoc
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
69–80
Language:
URL:
https://aclanthology.org/2021.dialdoc-1.10
DOI:
10.18653/v1/2021.dialdoc-1.10
Bibkey:
Cite (ACL):
Anant Khandelwal. 2021. WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue. In Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021), pages 69–80, Online. Association for Computational Linguistics.
Cite (Informal):
WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue (Khandelwal, dialdoc 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/2021.dialdoc-1.10.pdf
Data
DailyDialogPERSONA-CHAT