AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain

Jimin Hong, TaeHee Kim, Hyesu Lim, Jaegul Choo


Abstract
During the fine-tuning phase of transfer learning, the pretrained vocabulary remains unchanged, while model parameters are updated. The vocabulary generated based on the pretrained data is suboptimal for downstream data when domain discrepancy exists. We propose to consider the vocabulary as an optimizable parameter, allowing us to update the vocabulary by expanding it with domain specific vocabulary based on a tokenization statistic. Furthermore, we preserve the embeddings of the added words from overfitting to downstream data by utilizing knowledge learned from a pretrained language model with a regularization term. Our method achieved consistent performance improvements on diverse domains (i.e., biomedical, computer science, news, and reviews).
Anthology ID:
2021.emnlp-main.385
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4692–4700
Language:
URL:
https://aclanthology.org/2021.emnlp-main.385
DOI:
10.18653/v1/2021.emnlp-main.385
Bibkey:
Cite (ACL):
Jimin Hong, TaeHee Kim, Hyesu Lim, and Jaegul Choo. 2021. AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4692–4700, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain (Hong et al., EMNLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-3/2021.emnlp-main.385.pdf
Video:
 https://preview.aclanthology.org/nschneid-patch-3/2021.emnlp-main.385.mp4
Code
 Jimin9401/avocado