I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling
Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, Jason Weston
Abstract
To quantify how well natural language understanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We show that: (i) our newly collected dataset is notably more effective at providing supervision for the dialogue contradiction detection task than existing NLI data including those aimed to cover the dialogue domain; (ii) Transformer models that explicitly hinge on utterance structures for dialogue contradiction detection are more robust and generalize well on both analysis and out-of-distribution dialogues than standard (unstructured) Transformers. We also show that our best contradiction detection model correlates well with human judgments and further provide evidence for its usage in both automatically evaluating and improving the consistency of state-of-the-art generative chatbots.- Anthology ID:
- 2021.acl-long.134
- Volume:
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Editors:
- Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
- Venues:
- ACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1699–1713
- Language:
- URL:
- https://aclanthology.org/2021.acl-long.134
- DOI:
- 10.18653/v1/2021.acl-long.134
- Cite (ACL):
- Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021. I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1699–1713, Online. Association for Computational Linguistics.
- Cite (Informal):
- I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (Nie et al., ACL-IJCNLP 2021)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2021.acl-long.134.pdf
- Data
- ANLI, Blended Skill Talk, ConvAI2, MultiNLI, SNLI, Wizard of Wikipedia