Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models

Minseok Choi, Kyunghyun Min, Jaegul Choo

Abstract
Pretrained language models memorize vast amounts of information, including private and copyrighted data, raising significant safety concerns. Retraining these models after excluding sensitive data is prohibitively expensive, making machine unlearning a viable, cost-effective alternative. Previous research has focused on machine unlearning for monolingual models, but we find that unlearning in one language does not necessarily transfer to others. This vulnerability makes models susceptible to low-resource language attacks, where sensitive information remains accessible in less dominant languages. This paper presents a pioneering approach to machine unlearning for multilingual language models, selectively erasing information across different languages while maintaining overall performance. Specifically, our method employs an adaptive unlearning scheme that assigns language-dependent weights to address different language performances of multilingual language models. Empirical results demonstrate the effectiveness of our framework compared to existing unlearning baselines, setting a new standard for secure and adaptable multilingual language models.
Anthology ID:
2024.findings-emnlp.630
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2024
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10732–10747
Language:
URL:
https://preview.aclanthology.org/icon-24-ingestion/2024.findings-emnlp.630/
DOI:
10.18653/v1/2024.findings-emnlp.630
Bibkey:
Cite (ACL):
Minseok Choi, Kyunghyun Min, and Jaegul Choo. 2024. Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10732–10747, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models (Choi et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/icon-24-ingestion/2024.findings-emnlp.630.pdf
PDF Search Fix data