Changing the Mind of Transformers for Topically-Controllable Language Generation

Haw-Shiuan Chang, Jiaming Yuan, Mohit Iyyer, Andrew McCallum

Abstract
Large Transformer-based language models can aid human authors by suggesting plausible continuations of text written so far. However, current interactive writing assistants do not allow authors to guide text generation in desired topical directions. To address this limitation, we design a framework that displays multiple candidate upcoming topics, of which a user can select a subset to guide the generation. Our framework consists of two components: (1) a method that produces a set of candidate topics by predicting the centers of word clusters in the possible continuations, and (2) a text generation model whose output adheres to the chosen topics. The training of both components is self-supervised, using only unlabeled text. Our experiments demonstrate that our topic options are better than those of standard clustering approaches, and our framework often generates fluent sentences related to the chosen topics, as judged by automated metrics and crowdsourced workers.
Anthology ID:
2021.eacl-main.223
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2601–2611
Language:
URL:
https://aclanthology.org/2021.eacl-main.223
DOI:
10.18653/v1/2021.eacl-main.223
Bibkey:
Cite (ACL):
Haw-Shiuan Chang, Jiaming Yuan, Mohit Iyyer, and Andrew McCallum. 2021. Changing the Mind of Transformers for Topically-Controllable Language Generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2601–2611, Online. Association for Computational Linguistics.
Cite (Informal):
Changing the Mind of Transformers for Topically-Controllable Language Generation (Chang et al., EACL 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/nodalida-main-page/2021.eacl-main.223.pdf
Code
 iesl/interactive_LM
PDF Search Code