Text-to-Text Pre-Training for Data-to-Text Tasks

Mihir Kale, Abhinav Rastogi


Abstract
We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5 (Raffel et al., 2019), enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternatives such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-ofdomain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.
Anthology ID:
2020.inlg-1.14
Volume:
Proceedings of the 13th International Conference on Natural Language Generation
Month:
December
Year:
2020
Address:
Dublin, Ireland
Editors:
Brian Davis, Yvette Graham, John Kelleher, Yaji Sripada
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
97–102
Language:
URL:
https://aclanthology.org/2020.inlg-1.14
DOI:
10.18653/v1/2020.inlg-1.14
Bibkey:
Cite (ACL):
Mihir Kale and Abhinav Rastogi. 2020. Text-to-Text Pre-Training for Data-to-Text Tasks. In Proceedings of the 13th International Conference on Natural Language Generation, pages 97–102, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Text-to-Text Pre-Training for Data-to-Text Tasks (Kale & Rastogi, INLG 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/2020.inlg-1.14.pdf
Code
 google-research-datasets/ToTTo +  additional community code
Data
MultiWOZToTToWebNLG