Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge

Daniel Mela, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas

Abstract
Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT.
Anthology ID:
2024.findings-acl.347
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5831–5847
Language:
URL:
https://preview.aclanthology.org/build-pipeline-with-new-library/2024.findings-acl.347/
DOI:
10.18653/v1/2024.findings-acl.347
Bibkey:
Cite (ACL):
Daniel Mela, Aitor Gonzalez-Agirre, Javier Hernando, and Marta Villegas. 2024. Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5831–5847, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge (Mela et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/build-pipeline-with-new-library/2024.findings-acl.347.pdf
PDF Search