Domain Adversarial Fine-Tuning as an Effective Regularizer
Giorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, Ion Androutsopoulos
Abstract
In Natural Language Processing (NLP), pretrained language models (LMs) that are transferred to downstream tasks have been recently shown to achieve state-of-the-art results. However, standard fine-tuning can degrade the general-domain representations captured during pretraining. To address this issue, we introduce a new regularization technique, AFTER; domain Adversarial Fine-Tuning as an Effective Regularizer. Specifically, we complement the task-specific loss used during fine-tuning with an adversarial objective. This additional loss term is related to an adversarial classifier, that aims to discriminate between in-domain and out-of-domain text representations. Indomain refers to the labeled dataset of the task at hand while out-of-domain refers to unlabeled data from a different domain. Intuitively, the adversarial classifier acts as a regularize which prevents the model from overfitting to the task-specific domain. Empirical results on various natural language understanding tasks show that AFTER leads to improved performance compared to standard fine-tuning.- Anthology ID:
- 2020.findings-emnlp.278
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Trevor Cohn, Yulan He, Yang Liu
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3103–3112
- Language:
- URL:
- https://aclanthology.org/2020.findings-emnlp.278
- DOI:
- 10.18653/v1/2020.findings-emnlp.278
- Cite (ACL):
- Giorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, and Ion Androutsopoulos. 2020. Domain Adversarial Fine-Tuning as an Effective Regularizer. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3103–3112, Online. Association for Computational Linguistics.
- Cite (Informal):
- Domain Adversarial Fine-Tuning as an Effective Regularizer (Vernikos et al., Findings 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2020.findings-emnlp.278.pdf
- Code
- GeorgeVern/AFTERV1.0
- Data
- AG News, CoLA, GLUE, MRPC, SST, SST-2