What Makes Good In-Context Examples for GPT-3?
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen
Abstract
GPT-3 has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its in-context learning abilities. Despite its success, we found that the empirical results of GPT-3 depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-3’s in-context learning capabilities. Inspired by the recent success of leveraging a retrieval module to augment neural networks, we propose to retrieve examples that are semantically-similar to a test query sample to formulate its corresponding prompt. Intuitively, the examples selected with such a strategy may serve as more informative inputs to unleash GPT-3’s power of text generation. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random selection baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (44.3% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset).- Anthology ID:
- 2022.deelio-1.10
- Volume:
- Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland and Online
- Editors:
- Eneko Agirre, Marianna Apidianaki, Ivan Vulić
- Venue:
- DeeLIO
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 100–114
- Language:
- URL:
- https://aclanthology.org/2022.deelio-1.10
- DOI:
- 10.18653/v1/2022.deelio-1.10
- Cite (ACL):
- Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
- Cite (Informal):
- What Makes Good In-Context Examples for GPT-3? (Liu et al., DeeLIO 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.deelio-1.10.pdf
- Data
- GLUE, IMDb Movie Reviews, MultiNLI, Natural Questions, SNLI, SST, SST-2, TriviaQA