Investigating the Representation of Open Domain Dialogue Context for Transformer Models
Vishakh Padmakumar, Behnam Hedayatnia, Di Jin, Patrick Lange, Seokhwan Kim, Nanyun Peng, Yang Liu, Dilek Hakkani-Tur
Abstract
The bulk of work adapting transformer models to open-domain dialogue represents dialogue context as the concatenated set of turns in natural language. However, it is unclear if this is the best approach. In this work, we investigate this question by means of an empirical controlled experiment varying the dialogue context format from text-only formats (all recent utterances, summaries, selected utterances) as well as variants that are more structurally different (triples, AMR). We compare these formats based on fine-tuned model performance on two downstream tasks—knowledge selection and response generation. We find that simply concatenating the utterances works as a strong baseline in most cases, but is outperformed in longer contexts by a hybrid approach of combining a summary of the context with recent utterances. Through empirical analysis, our work highlights the need to examine the format of context representation and offers recommendations on adapting general-purpose language models to dialogue tasks.- Anthology ID:
- 2023.sigdial-1.50
- Volume:
- Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue
- Month:
- September
- Year:
- 2023
- Address:
- Prague, Czechia
- Editors:
- Svetlana Stoyanchev, Shafiq Joty, David Schlangen, Ondrej Dusek, Casey Kennington, Malihe Alikhani
- Venue:
- SIGDIAL
- SIG:
- SIGDIAL
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 538–547
- Language:
- URL:
- https://aclanthology.org/2023.sigdial-1.50
- DOI:
- 10.18653/v1/2023.sigdial-1.50
- Cite (ACL):
- Vishakh Padmakumar, Behnam Hedayatnia, Di Jin, Patrick Lange, Seokhwan Kim, Nanyun Peng, Yang Liu, and Dilek Hakkani-Tur. 2023. Investigating the Representation of Open Domain Dialogue Context for Transformer Models. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 538–547, Prague, Czechia. Association for Computational Linguistics.
- Cite (Informal):
- Investigating the Representation of Open Domain Dialogue Context for Transformer Models (Padmakumar et al., SIGDIAL 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-07/2023.sigdial-1.50.pdf