Abstract
Language modeling with BERT consists of two phases of (i) unsupervised pre-training on unlabeled text, and (ii) fine-tuning for a specific supervised task. We present a method that leverages the second phase to its fullest, by applying an extensive number of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during training. We conduct an extensive inter- and intra-dataset evaluation, showing that our method improves the generalization ability of BERT, sometimes leading to a +9% gain in accuracy. These results highlight the importance of a proper fine-tuning procedure, especially for relatively smaller-sized datasets. Our code is attached as supplementary.- Anthology ID:
- 2021.eacl-main.14
- Volume:
- Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
- Month:
- April
- Year:
- 2021
- Address:
- Online
- Editors:
- Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 187–199
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2021.eacl-main.14/
- DOI:
- 10.18653/v1/2021.eacl-main.14
- Cite (ACL):
- Itzik Malkiel and Lior Wolf. 2021. Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 187–199, Online. Association for Computational Linguistics.
- Cite (Informal):
- Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models (Malkiel & Wolf, EACL 2021)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2021.eacl-main.14.pdf
- Data
- CoLA, GLUE, MultiNLI, QNLI, SNLI, SST, SST-2