Abstract
To be good conversational partners, natural language processing (NLP) systems should be trained to produce contextually useful utterances. Prior work has investigated training NLP systems with communication-based objectives, where a neural listener stands in as a communication partner. However, these systems commonly suffer from semantic drift where the learned language diverges radically from natural language. We propose a method that uses a population of neural listeners to regularize speaker training. We first show that language drift originates from the poor uncertainty calibration of a neural listener, which makes high-certainty predictions on novel sentences. We explore ensemble- and dropout-based populations of listeners and find that the former results in better uncertainty quantification. We evaluate both population-based objectives on reference games, and show that the ensemble method with better calibration enables the speaker to generate pragmatic utterances while scaling to a large vocabulary and generalizing to new games and listeners.- Anthology ID:
- 2021.findings-emnlp.83
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2021
- Month:
- November
- Year:
- 2021
- Address:
- Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- Findings
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 977–984
- Language:
- URL:
- https://aclanthology.org/2021.findings-emnlp.83
- DOI:
- 10.18653/v1/2021.findings-emnlp.83
- Cite (ACL):
- Rose Wang, Julia White, Jesse Mu, and Noah Goodman. 2021. Calibrate your listeners! Robust communication-based training for pragmatic speakers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 977–984, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Calibrate your listeners! Robust communication-based training for pragmatic speakers (Wang et al., Findings 2021)
- PDF:
- https://preview.aclanthology.org/alta-23-ingestion/2021.findings-emnlp.83.pdf
- Code
- rosewang2008/calibrate_your_listeners
- Data
- ShapeWorld