Improving Text-to-Text Pre-trained Models for the Graph-to-Text Task

Zixiaofan Yang, Arash Einolghozati, Hakan Inan, Keith Diedrick, Angela Fan, Pinar Donmez, Sonal Gupta

Abstract
Converting a knowledge graph or sub-graph to natural text is useful when answering questions based on a knowledge base. High-capacity language models pre-trained on large-scale text corpora have recently been shown to be powerful when fine-tuned for the knowledge-graph-to-text (KG-to-text) task. In this paper, we propose two classes of methods to improve such pre-trained models for this task. First, we improve the structure awareness of the model by organizing the input as well as learning optimal ordering via multitask learning. Second, we bridge the domain gap between text-to-text and KG-to-text tasks via a second-phase KG-to-text pre-training on similar datasets and extra lexicalization supervision to make the input more similar to natural text. We demonstrate the efficacy of our methods on the popular WebNLG dataset. Our best model achieves an almost 3 point BLEU improvement on a strong baseline while lowering the relative slot-error-rate by around 35%. We also validate our results via human evaluation.
Anthology ID:
2020.webnlg-1.11
Volume:
Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)
Month:
12
Year:
2020
Address:
Dublin, Ireland (Virtual)
Editors:
Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, Anastasia Shimorina
Venue:
WebNLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
107–116
Language:
URL:
https://aclanthology.org/2020.webnlg-1.11
DOI:
Bibkey:
Cite (ACL):
Zixiaofan Yang, Arash Einolghozati, Hakan Inan, Keith Diedrick, Angela Fan, Pinar Donmez, and Sonal Gupta. 2020. Improving Text-to-Text Pre-trained Models for the Graph-to-Text Task. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 107–116, Dublin, Ireland (Virtual). Association for Computational Linguistics.
Cite (Informal):
Improving Text-to-Text Pre-trained Models for the Graph-to-Text Task (Yang et al., WebNLG 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2020.webnlg-1.11.pdf
Data
DocRED, WebNLG
PDF Search