@inproceedings{ko-li-2020-assessing,
    title = "Assessing Discourse Relations in Language Generation from {GPT}-2",
    author = "Ko, Wei-Jen  and
      Li, Junyi Jessy",
    editor = "Davis, Brian  and
      Graham, Yvette  and
      Kelleher, John  and
      Sripada, Yaji",
    booktitle = "Proceedings of the 13th International Conference on Natural Language Generation",
    month = dec,
    year = "2020",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2020.inlg-1.8/",
    doi = "10.18653/v1/2020.inlg-1.8",
    pages = "52--59",
    abstract = "Recent advances in NLP have been attributed to the emergence of large-scale pre-trained language models. GPT-2, in particular, is suited for generation tasks given its left-to-right language modeling objective, yet the linguistic quality of its generated text has largely remain unexplored. Our work takes a step in understanding GPT-2{'}s outputs in terms of discourse coherence. We perform a comprehensive study on the validity of explicit discourse relations in GPT-2{'}s outputs under both organic generation and fine-tuned scenarios. Results show GPT-2 does not always generate text containing valid discourse relations; nevertheless, its text is more aligned with human expectation in the fine-tuned scenario. We propose a decoupled strategy to mitigate these problems and highlight the importance of explicitly modeling discourse information."
}