Investigating Pretrained Language Models for Graph-to-Text Generation
Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, Iryna Gurevych
Abstract
Graph-to-text generation aims to generate fluent texts from graph-based data. In this paper, we investigate two recent pretrained language models (PLMs) and analyze the impact of different task-adaptive pretraining strategies for PLMs in graph-to-text generation. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that approaches based on PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further. We report new state-of-the-art BLEU scores of 49.72 on AMR-LDC2017T10, 59.70 on WebNLG, and 25.66 on AGENDA datasets - a relative improvement of 31.8%, 4.5%, and 42.4%, respectively, with our models generating significantly more fluent texts than human references. In an extensive analysis, we identify possible reasons for the PLMs’ success on graph-to-text tasks. Our findings suggest that the PLMs benefit from similar facts seen during pretraining or fine-tuning, such that they perform well even when the input graph is reduced to a simple bag of node and edge labels.- Anthology ID:
- 2021.nlp4convai-1.20
- Volume:
- Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI
- Month:
- November
- Year:
- 2021
- Address:
- Online
- Editors:
- Alexandros Papangelis, Paweł Budzianowski, Bing Liu, Elnaz Nouri, Abhinav Rastogi, Yun-Nung Chen
- Venue:
- NLP4ConvAI
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 211–227
- Language:
- URL:
- https://aclanthology.org/2021.nlp4convai-1.20
- DOI:
- 10.18653/v1/2021.nlp4convai-1.20
- Cite (ACL):
- Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating Pretrained Language Models for Graph-to-Text Generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 211–227, Online. Association for Computational Linguistics.
- Cite (Informal):
- Investigating Pretrained Language Models for Graph-to-Text Generation (Ribeiro et al., NLP4ConvAI 2021)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2021.nlp4convai-1.20.pdf
- Code
- UKPLab/plms-graph2text + additional community code
- Data
- AGENDA, DART, DBpedia, LDC2017T10, Semantic Scholar, WebNLG