@inproceedings{deng-etal-2022-model,
    title = "Model Criticism for Long-Form Text Generation",
    author = "Deng, Yuntian  and
      Kuleshov, Volodymyr  and
      Rush, Alexander",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2022.emnlp-main.815/",
    doi = "10.18653/v1/2022.emnlp-main.815",
    pages = "11887--11912",
    abstract = "Language models have demonstrated the ability to generate highly fluent text; however, it remains unclear whether their output retains coherent high-level structure (e.g., story progression). Here, we propose to apply a statistical tool, model criticism in latent space, to evaluate the high-level structure of the generated text. Model criticism compares the distributions between real and generated data in a latent space obtained according to an assumptive generative process. Different generative processes identify specific failure modes of the underlying model. We perform experiments on three representative aspects of high-level discourse{---}coherence, coreference, and topicality{---}and find that transformer-based language models are able to capture topical structures but have a harder time maintaining structural coherence or modeling coreference."
}