@inproceedings{padmakumar-etal-2023-investigating,
    title = "Investigating the Representation of Open Domain Dialogue Context for Transformer Models",
    author = "Padmakumar, Vishakh  and
      Hedayatnia, Behnam  and
      Jin, Di  and
      Lange, Patrick  and
      Kim, Seokhwan  and
      Peng, Nanyun  and
      Liu, Yang  and
      Hakkani-Tur, Dilek",
    editor = "Stoyanchev, Svetlana  and
      Joty, Shafiq  and
      Schlangen, David  and
      Dusek, Ondrej  and
      Kennington, Casey  and
      Alikhani, Malihe",
    booktitle = "Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue",
    month = sep,
    year = "2023",
    address = "Prague, Czechia",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2023.sigdial-1.50/",
    doi = "10.18653/v1/2023.sigdial-1.50",
    pages = "538--547",
    abstract = "The bulk of work adapting transformer models to open-domain dialogue represents dialogue context as the concatenated set of turns in natural language. However, it is unclear if this is the best approach. In this work, we investigate this question by means of an empirical controlled experiment varying the dialogue context format from text-only formats (all recent utterances, summaries, selected utterances) as well as variants that are more structurally different (triples, AMR). We compare these formats based on fine-tuned model performance on two downstream tasks{---}knowledge selection and response generation. We find that simply concatenating the utterances works as a strong baseline in most cases, but is outperformed in longer contexts by a hybrid approach of combining a summary of the context with recent utterances. Through empirical analysis, our work highlights the need to examine the format of context representation and offers recommendations on adapting general-purpose language models to dialogue tasks."
}