Abstract
Here we experiment with the use of information retrieval as an augmentation for pre-trained language models. The text corpus used in information retrieval can be viewed as form of episodic memory which grows over time. By augmenting GPT 2.0 with information retrieval we achieve a zero shot 15% relative reduction in perplexity on Gigaword corpus without any re-training. We also validate our IR augmentation on an event co-reference task.- Anthology ID:
- 2020.nuse-1.14
- Volume:
- Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Claire Bonial, Tommaso Caselli, Snigdha Chaturvedi, Elizabeth Clark, Ruihong Huang, Mohit Iyyer, Alejandro Jaimes, Heng Ji, Lara J. Martin, Ben Miller, Teruko Mitamura, Nanyun Peng, Joel Tetreault
- Venues:
- NUSE | WNU
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 114–119
- Language:
- URL:
- https://aclanthology.org/2020.nuse-1.14
- DOI:
- 10.18653/v1/2020.nuse-1.14
- Cite (ACL):
- Hai Wang and David McAllester. 2020. On-The-Fly Information Retrieval Augmentation for Language Models. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 114–119, Online. Association for Computational Linguistics.
- Cite (Informal):
- On-The-Fly Information Retrieval Augmentation for Language Models (Wang & McAllester, NUSE-WNU 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.nuse-1.14.pdf
- Data
- ECB+