Abstract
Generative neural conversational systems are typically trained by minimizing the entropy loss between the training “hard” targets and the predicted logits. Performance gains and improved generalization are often achieved by employing regularization techniques like label smoothing, which converts the training “hard” targets to soft targets. However, label smoothing enforces a data independent uniform distribution on the incorrect training targets, leading to a false assumption of equiprobability. In this paper, we propose and experiment with incorporating data-dependent word similarity-based weighing methods to transform the uniform distribution of the incorrect target probabilities in label smoothing to a more realistic distribution based on semantics. We introduce hyperparameters to control the incorrect target distribution and report significant performance gains over networks trained using standard label smoothing-based loss on two standard open-domain dialogue corpora.- Anthology ID:
- 2022.icon-main.31
- Volume:
- Proceedings of the 19th International Conference on Natural Language Processing (ICON)
- Month:
- December
- Year:
- 2022
- Address:
- New Delhi, India
- Editors:
- Md. Shad Akhtar, Tanmoy Chakraborty
- Venue:
- ICON
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 253–259
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.icon-main.31/
- DOI:
- Cite (ACL):
- Sougata Saha, Souvik Das, and Rohini Srihari. 2022. Similarity Based Label Smoothing For Dialogue Generation. In Proceedings of the 19th International Conference on Natural Language Processing (ICON), pages 253–259, New Delhi, India. Association for Computational Linguistics.
- Cite (Informal):
- Similarity Based Label Smoothing For Dialogue Generation (Saha et al., ICON 2022)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.icon-main.31.pdf