RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
Yasuto Hoshi, Daisuke Miyashita, Youyang Ng, Kento Tatsuno, Yasuhiro Morioka, Osamu Torii, Jun Deguchi
Abstract
Retrieval-augmented large language models (R-LLMs) combine pre-trained large language models (LLMs) with information retrieval systems to improve the accuracy of factual question-answering. However, current libraries for building R-LLMs provide high-level abstractions without sufficient transparency for evaluating and optimizing prompts within specific inference processes such as retrieval and generation. To address this gap, we present RaLLe, an open-source framework designed to facilitate the development, evaluation, and optimization of R-LLMs for knowledge-intensive tasks. With RaLLe, developers can easily develop and evaluate R-LLMs, improving hand-crafted prompts, assessing individual inference processes, and objectively measuring overall system performance quantitatively. By leveraging these features, developers can enhance the performance and accuracy of their R-LLMs in knowledge-intensive generation tasks.- Anthology ID:
- 2023.emnlp-demo.4
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Yansong Feng, Els Lefever
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 52–69
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-demo.4
- DOI:
- 10.18653/v1/2023.emnlp-demo.4
- Cite (ACL):
- Yasuto Hoshi, Daisuke Miyashita, Youyang Ng, Kento Tatsuno, Yasuhiro Morioka, Osamu Torii, and Jun Deguchi. 2023. RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 52–69, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models (Hoshi et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/insights-reingestion/2023.emnlp-demo.4.pdf