Abstract
Large language models show an emergent ability to learn a new task from a small number of input-output demonstrations. However, recent work shows that in-context learners largely rely on their pre-trained knowledge, such as the sentiment of the labels, instead of finding new associations in the input. However, the commonly-used few-shot evaluation settings using a random selection of in-context demonstrations can not disentangle models’ ability to learn a new skill from demonstrations, as most of the randomly-selected demonstrations do not present relations informative for prediction beyond exposing the new task distribution. To disentangle models’ in-context learning ability independent of models’ memory, we introduce a Conceptual few-shot learning method selecting the demonstrations sharing a possibly-informative concept with the predicted sample. We extract a set of such concepts from annotated explanations and measure how much can models benefit from presenting these concepts in few-shot demonstrations. We find that smaller models are more sensitive to the presented concepts. While some of the models are able to benefit from concept-presenting demonstrations for each assessed concept, we find that none of the assessed in-context learners can benefit from all presented reasoning concepts consistently, leaving the in-context concept learning an open challenge.- Anthology ID:
- 2023.nlrse-1.8
- Volume:
- Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE)
- Month:
- June
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Danilo Neves Ribeiro, Jason Wei
- Venue:
- NLRSE
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 107–115
- Language:
- URL:
- https://aclanthology.org/2023.nlrse-1.8
- DOI:
- 10.18653/v1/2023.nlrse-1.8
- Cite (ACL):
- Michal Štefánik and Marek Kadlčík. 2023. Can In-context Learners Learn a Reasoning Concept from Demonstrations?. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 107–115, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Can In-context Learners Learn a Reasoning Concept from Demonstrations? (Štefánik & Kadlčík, NLRSE 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.nlrse-1.8.pdf