Abstract
We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5 (Raffel et al., 2019), enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternatives such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-ofdomain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.- Anthology ID:
- 2020.inlg-1.14
- Volume:
- Proceedings of the 13th International Conference on Natural Language Generation
- Month:
- December
- Year:
- 2020
- Address:
- Dublin, Ireland
- Editors:
- Brian Davis, Yvette Graham, John Kelleher, Yaji Sripada
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 97–102
- Language:
- URL:
- https://aclanthology.org/2020.inlg-1.14
- DOI:
- 10.18653/v1/2020.inlg-1.14
- Cite (ACL):
- Mihir Kale and Abhinav Rastogi. 2020. Text-to-Text Pre-Training for Data-to-Text Tasks. In Proceedings of the 13th International Conference on Natural Language Generation, pages 97–102, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Text-to-Text Pre-Training for Data-to-Text Tasks (Kale & Rastogi, INLG 2020)
- PDF:
- https://preview.aclanthology.org/naacl-24-ws-corrections/2020.inlg-1.14.pdf
- Code
- google-research-datasets/ToTTo + additional community code
- Data
- MultiWOZ, ToTTo, WebNLG