Abstract
Elastic weight consolidation (EWC, Kirkpatrick et al. 2017) is a promising approach to addressing catastrophic forgetting in sequential training. We find that the effect of EWC can diminish when fine-tuning large-scale pre-trained language models on different datasets. We present two simple objective functions to mitigate this problem by rescaling the components of EWC. Experiments on natural language inference and fact-checking tasks indicate that our methods require much smaller values for the trade-off parameters to achieve results comparable to EWC.- Anthology ID:
- 2022.coling-1.403
- Volume:
- Proceedings of the 29th International Conference on Computational Linguistics
- Month:
- October
- Year:
- 2022
- Address:
- Gyeongju, Republic of Korea
- Editors:
- Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 4568–4574
- Language:
- URL:
- https://aclanthology.org/2022.coling-1.403
- DOI:
- Cite (ACL):
- Canasai Kruengkrai and Junichi Yamagishi. 2022. Mitigating the Diminishing Effect of Elastic Weight Consolidation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4568–4574, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Cite (Informal):
- Mitigating the Diminishing Effect of Elastic Weight Consolidation (Kruengkrai & Yamagishi, COLING 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.coling-1.403.pdf
- Data
- FEVER, MultiNLI