When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, Jing Xu

Abstract
Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.
Anthology ID:
2024.naacl-long.169
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3066–3082
Language:
URL:
https://aclanthology.org/2024.naacl-long.169
DOI:
10.18653/v1/2024.naacl-long.169
Bibkey:
Cite (ACL):
Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. 2024. When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3066–3082, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (Shi et al., NAACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.naacl-long.169.pdf
PDF Search