DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models

Shanbao Qiao, Xuebing Liu, Seung-Hoon Na

Abstract
Among the recently emerged knowledge editing methods, in-context knowledge editing (IKE) has shown respectable abilities on knowledge editing in terms of generalization and specificity. Noting the promising advantages but unexplored issues of IKE, we propose **DistillMIKE** as a novel extension of IKE, i.e., editing **distill**ation of "**M**assive” **I**n-context **K**nowledge **E**diting in large language models (LLMs), mainly consisting of two expansions; 1) *Massive in-context knowledge editing (MIKE)*, which extends IKE to a massive editing task, aiming to inject not a single edit but a set of massive edits to LLMs; To preserve specificity, our key novel extension is a “selective” retrieval augmentation, where the retrieval-augmented IKE is only applied to “in-scope” examples, whereas the unedited model without IKE is employed for “out-of-scope” ones. 2) *Editing distillation* of MIKE using low-rank adaptation (LoRA), which distills editing abilities of MIKE to parameters of LLMs in a manner of eliminating the need of lengthy in-context demonstrations, thus removing the computational overhead encountered at the inference time. Experimental results on the zsRE and CounterFact datasets demonstrate that MIKE shows the state-of-the-art perfomrances and DistilMIKE show comparable performances with MIKE. Our code is available at https://github.com/JoveReCode/DistillMIKE.git.
Anthology ID:
2024.findings-acl.455
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7639–7654
Language:
URL:
https://preview.aclanthology.org/build-pipeline-with-new-library/2024.findings-acl.455/
DOI:
10.18653/v1/2024.findings-acl.455
Bibkey:
Cite (ACL):
Shanbao Qiao, Xuebing Liu, and Seung-Hoon Na. 2024. DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7639–7654, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models (Qiao et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/build-pipeline-with-new-library/2024.findings-acl.455.pdf
PDF Search