Abstract
We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.- Anthology ID:
- 2020.emnlp-main.493
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6102–6120
- Language:
- URL:
- https://preview.aclanthology.org/remove-affiliations/2020.emnlp-main.493/
- DOI:
- 10.18653/v1/2020.emnlp-main.493
- Cite (ACL):
- Minki Kang, Moonsu Han, and Sung Ju Hwang. 2020. Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6102–6120, Online. Association for Computational Linguistics.
- Cite (Informal):
- Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation (Kang et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/remove-affiliations/2020.emnlp-main.493.pdf
- Code
- Nardien/NMG
- Data
- IMDb Movie Reviews, NewsQA, emrQA