Generating Text from Language Models
Afra Amini, Ryan Cotterell, John Hewitt, Luca Malagutti, Clara Meister, Tiago Pimentel
Abstract
An increasingly large percentage of natural language processing (NLP) tasks center around the generation of text from probabilistic language models. Despite this trend, techniques for improving or specifying preferences in these generated texts rely mostly on intuition-based heuristics. Further, there lacks a unified presentation of their motivations, practical implementation, successes and pitfalls. Practitioners must, therefore, choose somewhat blindly between generation algorithms—like top-p sampling or beam search—which can lead to wildly different results. At the same time, language generation research continues to criticize and improve the standard toolboxes, further adding entropy to the state of the field. In this tutorial, we will provide a centralized and cohesive discussion of critical considerations when choosing how to generate from a language model. We will cover a wide range of empirically-observed problems (like degradation, hallucination, repetition) and their corresponding proposed algorithmic solutions from recent research (like top-p sampling and its successors). We will then discuss a subset of these algorithms under a unified light; most stochastic generation strategies can be framed as locally adapting the probabilities of a model to avoid failure cases. Finally, we will then cover methods in controlled generation, that go beyond just ensuring coherence to ensure text exhibits specific desired properties. We aim for NLP practitioners and researchers to leave our tutorial with a unified framework which they can use to evaluate and contribute to the latest research in language generation.- Anthology ID:
- 2023.acl-tutorials.4
- Original:
- 2023.acl-tutorials.4v1
- Version 2:
- 2023.acl-tutorials.4v2
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Yun-Nung (Vivian) Chen, Margot Margot, Siva Reddy
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 27–31
- Language:
- URL:
- https://aclanthology.org/2023.acl-tutorials.4
- DOI:
- 10.18653/v1/2023.acl-tutorials.4
- Cite (ACL):
- Afra Amini, Ryan Cotterell, John Hewitt, Luca Malagutti, Clara Meister, and Tiago Pimentel. 2023. Generating Text from Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 27–31, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Generating Text from Language Models (Amini et al., ACL 2023)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2023.acl-tutorials.4.pdf