Controllable Sentence Simplification with a Unified Text-to-Text Transfer Transformer

Kim Cheng Sheang, Horacio Saggion


Abstract
Recently, a large pre-trained language model called T5 (A Unified Text-to-Text Transfer Transformer) has achieved state-of-the-art performance in many NLP tasks. However, no study has been found using this pre-trained model on Text Simplification. Therefore in this paper, we explore the use of T5 fine-tuning on Text Simplification combining with a controllable mechanism to regulate the system outputs that can help generate adapted text for different target audiences. Our experiments show that our model achieves remarkable results with gains of between +0.69 and +1.41 over the current state-of-the-art (BART+ACCESS). We argue that using a pre-trained model such as T5, trained on several tasks with large amounts of data, can help improve Text Simplification.
Anthology ID:
2021.inlg-1.38
Volume:
Proceedings of the 14th International Conference on Natural Language Generation
Month:
August
Year:
2021
Address:
Aberdeen, Scotland, UK
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
341–352
Language:
URL:
https://aclanthology.org/2021.inlg-1.38
DOI:
Bibkey:
Cite (ACL):
Kim Cheng Sheang and Horacio Saggion. 2021. Controllable Sentence Simplification with a Unified Text-to-Text Transfer Transformer. In Proceedings of the 14th International Conference on Natural Language Generation, pages 341–352, Aberdeen, Scotland, UK. Association for Computational Linguistics.
Cite (Informal):
Controllable Sentence Simplification with a Unified Text-to-Text Transfer Transformer (Sheang & Saggion, INLG 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2021.inlg-1.38.pdf
Code
 kimchengsheang/ts_t5
Data
ASSETTurkCorpusWikiLarge