Abstract
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model and a knowledge graph for more coherent and logical generation.- Anthology ID:
- 2022.tacl-1.32
- Volume:
- Transactions of the Association for Computational Linguistics, Volume 10
- Month:
- Year:
- 2022
- Address:
- Cambridge, MA
- Venue:
- TACL
- SIG:
- Publisher:
- MIT Press
- Note:
- Pages:
- 555–572
- Language:
- URL:
- https://aclanthology.org/2022.tacl-1.32
- DOI:
- 10.1162/tacl_a_00476
- Cite (ACL):
- Qi Liu, Dani Yogatama, and Phil Blunsom. 2022. Relational Memory-Augmented Language Models. Transactions of the Association for Computational Linguistics, 10:555–572.
- Cite (Informal):
- Relational Memory-Augmented Language Models (Liu et al., TACL 2022)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2022.tacl-1.32.pdf