Richard James


2024

pdf
REPLUG: Retrieval-Augmented Black-Box Language Models
Weijia Shi | Sewon Min | Michihiro Yasunaga | Minjoon Seo | Richard James | Mike Lewis | Luke Zettlemoyer | Wen-tau Yih
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross-attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by 5.1%. Code is publicly released at github.com/swj0419/REPLUG.

2023

pdf
Reimagining Retrieval Augmented Language Models for Answering Queries
Wang-Chiew Tan | Yuliang Li | Pedro Rodriguez | Richard James | Xi Victoria Lin | Alon Halevy | Wen-tau Yih
Findings of the Association for Computational Linguistics: ACL 2023

We present a reality check on large language models and inspect the promise of retrieval-augmented language models in comparison. Such language models are semi-parametric, where models integrate model parameters and knowledge from external data sources to make their predictions, as opposed to the parametric nature of vanilla large language models. We give initial experimental findings that semi-parametric architectures can be enhanced with views, a query analyzer/planner, and provenance to make a significantly more powerful system for question answering in terms of accuracy and efficiency, and potentially for other NLP tasks.