Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token
Baohao Liao, David Thulke, Sanjika Hewavitharana, Hermann Ney, Christof Monz
Abstract
The pre-training of masked language models (MLMs) consumes massive computation to achieve good results on downstream NLP tasks, resulting in a large carbon footprint. In the vanilla MLM, the virtual tokens, [MASK]s, act as placeholders and gather the contextualized information from unmasked tokens to restore the corrupted information. It raises the question of whether we can append [MASK]s at a later layer, to reduce the sequence length for earlier layers and make the pre-training more efficient. We show: (1) [MASK]s can indeed be appended at a later layer, being disentangled from the word embedding; (2) The gathering of contextualized information from unmasked tokens can be conducted with a few layers. By further increasing the masking rate from 15% to 50%, we can pre-train RoBERTa-base and RoBERTa-large from scratch with only 78% and 68% of the original computational budget without any degradation on the GLUE benchmark. When pre-training with the original budget, our method outperforms RoBERTa for 6 out of 8 GLUE tasks, on average by 0.4%.- Anthology ID:
- 2022.findings-emnlp.106
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1478–1492
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.106
- DOI:
- 10.18653/v1/2022.findings-emnlp.106
- Cite (ACL):
- Baohao Liao, David Thulke, Sanjika Hewavitharana, Hermann Ney, and Christof Monz. 2022. Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1478–1492, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token (Liao et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2022.findings-emnlp.106.pdf