Abstract
Practical dialogue systems require robust methods of detecting out-of-scope (OOS) utterances to avoid conversational breakdowns and related failure modes. Directly training a model with labeled OOS examples yields reasonable performance, but obtaining such data is a resource-intensive process. To tackle this limited-data problem, previous methods focus on better modeling the distribution of in-scope (INS) examples. We introduce GOLD as an orthogonal technique that augments existing data to train better OOS detectors operating in low-data regimes. GOLD generates pseudo-labeled candidates using samples from an auxiliary dataset and keeps only the most beneficial candidates for training through a novel filtering mechanism. In experiments across three target benchmarks, the top GOLD model outperforms all existing methods on all key metrics, achieving relative gains of 52.4%, 48.9% and 50.3% against median baseline performance. We also analyze the unique properties of OOS data to identify key factors for optimally applying our proposed method.- Anthology ID:
- 2021.emnlp-main.35
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 429–442
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.35
- DOI:
- 10.18653/v1/2021.emnlp-main.35
- Cite (ACL):
- Derek Chen and Zhou Yu. 2021. GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 429–442, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentation (Chen & Yu, EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2021.emnlp-main.35.pdf
- Code
- asappresearch/gold
- Data
- ROSTD, STAR