Text Revision by On-the-Fly Representation Optimization

Jingjing Li, Zichao Li, Tao Ge, Irwin King, Michael Lyu


Abstract
Text revision refers to a family of natural language generation tasks, where the source and target sequences share moderate resemblance in surface form but differentiate in attributes, such as text formality and simplicity. Current state-of-the-art methods formulate these tasks as sequence-to-sequence learning problems, which rely on large-scale parallel training corpus. In this paper, we present an iterative inplace editing approach for text revision, which requires no parallel data. In this approach, we simply fine-tune a pre-trained Transformer with masked language modeling and attribute classification. During inference, the editing at each iteration is realized by two-step span replacement. At the first step, the distributed representation of the text optimizes on the fly towards an attribute function. At the second step, a text span is masked and another new one is proposed conditioned on the optimized representation. The empirical experiments on two typical and important text revision tasks, text formalization and text simplification, show the effectiveness of our approach. It achieves competitive and even better performance than state-of-the-art supervised methods on text simplification, and gains better performance than strong unsupervised methods on text formalization.
Anthology ID:
2022.in2writing-1.7
Volume:
Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
In2Writing
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
58–59
Language:
URL:
https://aclanthology.org/2022.in2writing-1.7
DOI:
10.18653/v1/2022.in2writing-1.7
Bibkey:
Cite (ACL):
Jingjing Li, Zichao Li, Tao Ge, Irwin King, and Michael Lyu. 2022. Text Revision by On-the-Fly Representation Optimization. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 58–59, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Text Revision by On-the-Fly Representation Optimization (Li et al., In2Writing 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.in2writing-1.7.pdf
Video:
 https://preview.aclanthology.org/ingestion-script-update/2022.in2writing-1.7.mp4
Code
 jingjingli01/oreo
Data
GYAFCNewsela