LM-BFF-MS: Improving Few-Shot Fine-tuning of Language Models based on Multiple Soft Demonstration Memory
Eunhwan Park, Donghyeon Jeon, Seonhoon Kim, Inho Kang, Seung-Hoon Na
Abstract
LM-BFF (CITATION) achieves significant few-shot performance by using auto-generated prompts and adding demonstrations similar to an input example. To improve the approach of LM-BFF, this paper proposes LM-BFF-MS—better few-shot fine-tuning of language models with multiple soft demonstrations by making its further extensions, which include 1) prompts with multiple demonstrations based on automatic generation of multiple label words; and 2) soft demonstration memory which consists of multiple sequences of globally shared word embeddings for a similar context. Experiments conducted on eight NLP tasks show that LM-BFF-MS leads to improvements over LM-BFF on five tasks, particularly achieving 94.0 and 90.4 on SST-2 and MRPC, respectively.- Anthology ID:
- 2022.acl-short.34
- Volume:
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Smaranda Muresan, Preslav Nakov, Aline Villavicencio
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 310–317
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2022.acl-short.34/
- DOI:
- 10.18653/v1/2022.acl-short.34
- Cite (ACL):
- Eunhwan Park, Donghyeon Jeon, Seonhoon Kim, Inho Kang, and Seung-Hoon Na. 2022. LM-BFF-MS: Improving Few-Shot Fine-tuning of Language Models based on Multiple Soft Demonstration Memory. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 310–317, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- LM-BFF-MS: Improving Few-Shot Fine-tuning of Language Models based on Multiple Soft Demonstration Memory (Park et al., ACL 2022)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2022.acl-short.34.pdf
- Code
- judepark96/lm-bff-ms
- Data
- MPQA Opinion Corpus, MRPC, MultiNLI, SNLI, SST, SST-2