The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank
Daphne Ippolito, Liam Dugan, Emily Reif, Ann Yuan, Andy Coenen, Chris Callison-Burch
Abstract
The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do fill in the blank, a more useful model is one that can effectively perform _both_ FitB and continuation tasks. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how these models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.- Anthology ID:
- 2022.findings-naacl.185
- Volume:
- Findings of the Association for Computational Linguistics: NAACL 2022
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, United States
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2421–2432
- Language:
- URL:
- https://aclanthology.org/2022.findings-naacl.185
- DOI:
- 10.18653/v1/2022.findings-naacl.185
- Cite (ACL):
- Daphne Ippolito, Liam Dugan, Emily Reif, Ann Yuan, Andy Coenen, and Chris Callison-Burch. 2022. The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2421–2432, Seattle, United States. Association for Computational Linguistics.
- Cite (Informal):
- The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank (Ippolito et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.findings-naacl.185.pdf
- Data
- WritingPrompts