Abstract
Intent Classification (IC) and Slot Labeling (SL) models, which form the basis of dialogue systems, often encounter noisy data in real-word environments. In this work, we investigate how robust IC/SL models are to noisy data. We collect and publicly release a test-suite for seven common noise types found in production human-to-bot conversations (abbreviations, casing, misspellings, morphological variants, paraphrases, punctuation and synonyms). On this test-suite, we show that common noise types substantially degrade the IC accuracy and SL F1 performance of state-of-the-art BERT-based IC/SL models. By leveraging cross-noise robustness transfer, i.e. training on one noise type to improve robustness on another noise type, we design aggregate data-augmentation approaches that increase the model performance across all seven noise types by +10.8% for IC accuracy and +15 points for SL F1 on average. To the best of our knowledge, this is the first work to present a single IC/SL model that is robust to a wide range of noise phenomena.- Anthology ID:
- 2021.nlp4convai-1.7
- Volume:
- Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI
- Month:
- November
- Year:
- 2021
- Address:
- Online
- Editors:
- Alexandros Papangelis, Paweł Budzianowski, Bing Liu, Elnaz Nouri, Abhinav Rastogi, Yun-Nung Chen
- Venue:
- NLP4ConvAI
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 68–79
- Language:
- URL:
- https://aclanthology.org/2021.nlp4convai-1.7
- DOI:
- 10.18653/v1/2021.nlp4convai-1.7
- Cite (ACL):
- Sailik Sengupta, Jason Krone, and Saab Mansour. 2021. On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 68–79, Online. Association for Computational Linguistics.
- Cite (Informal):
- On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise (Sengupta et al., NLP4ConvAI 2021)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2021.nlp4convai-1.7.pdf
- Code
- amazon-research/real-world-noisy-benchmarks-for-natural-language-understanding
- Data
- ATIS