%
% File acl2021.tex
%
%% Based on the style files for EMNLP 2020, which were
%% Based on the style files for ACL 2020, which were
%% Based on the style files for ACL 2018, NAACL 2018/19, which were
%% Based on the style files for ACL-2015, with some improvements
%%  taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by 
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith

\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{acl2021}
\special{papersize=210mm,297mm}

% pakages
\usepackage{paralist}
\usepackage{graphicx}
%\renewcommand\appendix{\end{document}}


\usepackage{times}
\usepackage{latexsym}
\usepackage{amsmath}
            
\renewcommand{\UrlFont}{\ttfamily\small}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

\aclfinalcopy % Uncomment this line for the final submission
\def\aclpaperid{30} %  Enter the acl Paper ID here

\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.

\newcommand\BibTeX{B\textsc{ib}\TeX}

\title{Revisiting Pretraining with Adapters}

\author{Seungwon Kim, Alex Shum, Nathan Susanj, Jonathan Hilgart \\
Georgia Institute of Technology \\
\texttt{\{skim3222, ashum7, nsusanj3, jhilgart3\}@gatech.edu}\\
}
%\author{
%Seungwon Kim \\
%  Georgia Institute of Technology \\
%  \texttt{skim3222@gatech.edu} \\\And
%  Alex Shum \\
%  Georgia Institute of Technology \\
%  \texttt{ashum7@gatech.edu} \\\AND
%  Nathan Susanj \\
%  Georgia Institute of Technology \\
%  \texttt{nsusanj3@gatech.edu} \\\And
%  Jonathan Hilgart \\
%  Georgia Institute of Technology \\
%  \texttt{jhilgart3@gatech.edu}\\
%  } 


\date{}

\begin{document}
\maketitle
\begin{abstract}
Pretrained language models have served as the backbone for many state-of-the-art NLP results. These models are large and expensive to train. Recent work suggests that continued pretraining on task-specific data is worth the effort as pretraining leads to improved performance on downstream tasks. We explore alternatives to full-scale task-specific pretraining of language models through the use of adapter modules, a parameter-efficient approach to transfer learning. We find that adapter-based pretraining is able to achieve comparable results to task-specific pretraining while using a fraction of the overall trainable parameters. We further explore direct use of adapters without pretraining and find that the direct fine-tuning performs mostly on par with pretrained adapter models, contradicting previously proposed benefits of continual pretraining in full pretraining fine-tuning strategies. Lastly, we perform an ablation study on task-adaptive pretraining to investigate how different hyperparameter settings can change the effectiveness of the pretraining. 
\end{abstract}


\section{Introduction}
\label{sef:intro}
Pretrained Language Models (PLM) are predominant in tackling current Natural Language Processing (NLP) tasks. Most PLMs based on the Transformer architecture \citep{NIPS2017_3f5ee243} are first trained on massive text corpora with the self-supervised objective to learn word representations \citep{devlin-etal-2019-bert, liu2019roberta}, and then are fine-tuned for a specific target task. The pretraining and fine-tuning of PLMs achieves state-of-the-art (SOTA) performance in many NLP tasks. Inspired by the benefits of pretraining, there have been studies demonstrate the effects of continued pretraining on the domain of a target task or the target task dataset \citep{Mitra2019ExploringWT, han_eisenstein_emnlp_2019, gururangan-etal-2020-dont}. \citealp{gururangan-etal-2020-dont} adapt PLMs on the target task by further pretraining RoBERTa \citep{liu2019roberta} on the target text corpus before it is fine-tuned for the corresponding task and showed that this task adaptation consistently improves the performance for text classification tasks. 

However, this full process of pretraining and then fine-tuning can be parameter inefficient for recent PLMs that have millions or billions of parameters \citep{devlin-etal-2019-bert, radford2018improving}. This parameter inefficiency becomes even worse when one continues pre-training all the parameters of PLMs on the task-specific corpus. Furthermore, recent PLMs need more than 100s of MB to store all the weights \citep{liu2019roberta, radford2018improving}, making it difficult to download and share the pre-trained models on the fly. 

Recently, adapters have been proposed as an alternative approach to decrease the substantial number of parameters of PLMs in the fine-tuning stage \citep{pmlr-v97-houlsby19a}. Finetuning with adapters mostly matches the performance of those with the full fine-tuning strategy on many NLP tasks including GLUE benchmark \citep{wang-etal-2018-glue} and reduces the size of the model from 100s of MB to the order of MB \citep{pfeiffer-etal-2020-adapterhub}. As such, a natural question arises from the successes of the adapter approach: can the adapter alone adapt PLMs to the target task when it is used in the second phase of the pretraining stage and thus lead to the improvement of the performance on the corresponding task? 

In this paper, we explore task-adaptive pretraining, termed TAPT \citep{gururangan-etal-2020-dont}, with adapters to address this question and overcome the limitations of the conventional full pretraining and fine-tuning.  We only train the adapter modules in the second phase of pretraining as well as the fine-tuning stage to achieve both parameter efficiency and the benefits of continual pretraining and compare those with the adapter-based model without pretraining. Surprisingly, we find that directly fine-tuning adapters performs mostly on par with the pre-trained adapter model and outperforms the full TAPT, contradicting the previously proposed benefits of continual pretraining in the full pretraining fine-tuning scheme. As directly fine-tuning adapters skips the second phase of pretraining and the training steps of adapters are faster than those of the full model, it substantially reduces the training time. We further investigate different hyperparameter settings that affect the effectiveness of pretraining.

\section{Pretraining and Adapters}
\paragraph{Pre-trained language model} 
We use RoBERTa \citep{liu2019roberta}, a Transformer-based language model that is pre-trained on a massive text corpus, following \citealp{gururangan-etal-2020-dont}. RoBERTa is an extension of BERT \citep{devlin-etal-2019-bert} with optimized hyperparameters and a modification of the pretraining objective, which excludes next sentence prediction and only uses the randomly masked tokens in the input sentence. To evaluate the performance of RoBERTa on a certain task, a classification layer is appended on top of the language model after the pretraining and all the parameters in RoBERTa are trained in a supervised way using the label of the dataset. In this paper, training word representations using RoBERTa on a masked language modeling task will be referred to as pretraining. Further, taking this pretrained model and adding a classification layer with additional updates to the language model parameters will be referred to as fine-tuning.  

\paragraph{Task-adaptive pretraining (TAPT)} 
Although RoBERTa achieves strong performance by simply fine-tuning the PLMs on a target task, there can be a distributional mismatch between the pretraining and target corpora. To address this issue, pretraining on the target task or the domain of the target task can be usefully employed to adapt the language models to the target task and it further improves the performance of the PLMs. Such methods can be referred to as Domain-Adaptive Pretraining (DAPT) or Task Adaptive-Pretraining (TAPT) \citep{gururangan-etal-2020-dont}. In this paper, we limit the scope of our works to TAPT as domain text corpus is not always available for each task, whereas TAPT can be easily applied by directly using the dataset of the target task while its performance often matches with DAPT \citep{gururangan-etal-2020-dont}. In TAPT, the second phase of pretraining is performed with RoBERTa using the unlabeled text corpus of the target task, and then it is fine-tuned on the target task. 


\paragraph{Adapter} 
Adapter modules have been employed as a feature extractor in computer vision \citep{NIPS2017_e7b24b11} and have been recently adopted in the NLP literature as an alternative approach to fully fine-tuning PLMs. Adapters are sets of new weights that are typically embedded in each transformer layer of PLMs and consist of feed-forward layers with normalizations, residual connections, and projection layers. The architectures of adapters vary with respect to the different configuration settings. We use the configuration proposed by \citealp{pfeiffer2020adapterfusion} in Figure \ref{fig:Adapter_Architecture}, which turned out to be effective on diverse NLP tasks, and add the adapter layer to each transformer layer.


\begin{figure} 
\centering
\includegraphics[width=0.65\linewidth]{adapters.png}
\caption{The adapter achitecture in the Transformer layer \citep{pfeiffer2020adapterfusion}}
\label{fig:Adapter_Architecture}
\end{figure}


\begin{table*}
\centering
\small
\begin{tabular}{|l|l|l|l|l|}
\hline
Domain & Task & Label type & Number of inst (Train/Dev/Test) & Classes \\
\hline\hline
Biomedical & \textsc{CHEMPROT} & Relationship classification & 4169 / 2427 / 3469 & 13 \\
Biomedical & \textsc{RCT} & Abstract sentence roles & 18040 / 30212 / 30135 & 5\\
Computer Science & \textsc{ACL-ARC} & Citation intent & 1688 / 114 / 139 & 6\\
Computer Science & \textsc{SCIERC} & Relation classification & 3219 / 455 / 974 & 7\\
News & \textsc{HYPERPARTISAN} & Partisanship & 515 / 65 / 65 & 2\\
News & \textsc{AGNEWS} & Topic & 115000 / 5000 / 7600 & 4\\
Reviews & \textsc{HELPFULNESS} & Review helpfulness & 115251 / 5000 / 25000 & 2\\
Reviews & \textsc{IMDB} & Review sentiment & 20000 / 5000 / 25000 & 2\\
\hline
\end{tabular}
\centering
\caption{Datasets used for experimentation. Datasets include both high-resource (RCT \citep{dernoncourt-lee-2017-pubmed}, \textsc{AGNEWS} \citep{NIPS2015_250cf8b5}, \textsc{HELPFULNESS} \citep{mcauley2015image}, \textsc{IMDB} \citep{maas-etal-2011-learning}) and low-resource (\textsc{CHEMPROT} \citep{kringelum2016chemprot}, \textsc{ACL-ARC} \citep{jurgens-etal-2018-measuring}, \textsc{SCIERC} \citep{luan-etal-2018-multi}, \textsc{HYPERPARTISAN} \citep{kiesel-etal-2019-semeval} settings.}
\label{tab:data}
\end{table*}

\citealp{pfeiffer-etal-2020-mad} use two types of adapter: language-specific adapters and task-specific adapters for cross-lingual transfer. These two types of adapter modules have similar architecture as in Figure \ref{fig:Adapter_Architecture}. However, the language adapters involve invertible adapters after the embedding layer to capture token-level language representation when those are trained via masked language modeling in the pretraining stage, whereas the task adapters are simply embedded in each transformer layer and trained in the fine-tuning stage to learn the task representation. Following \citealp{pfeiffer-etal-2020-mad}, we employ language adapter modules with invertible layers to perform pretraining adapters on the unlabeled target dataset. However, we perform fine-tuning pre-trained parameters of the language adapter modules for evaluation to align with TAPT, whereas \citealp{pfeiffer-etal-2020-mad} employ both the language and the task adapters by stacking task adapters on top of the language adapters.

\section{Experiments}
\label{sec:tapa_e10}
We now propose an adapter-based approach that is a parameter efficient variant of Task-Adaptive Pretraining (TAPT) and measure the margin of the performance between the pre-trained adapter model and the adapter model without pretraining. For pretraining adapters, we added the adapter module in each transformer layer of RoBERTa using adapter-transformer \citep{pfeiffer-etal-2020-adapterhub}\footnote{\url{https://github.com/Adapter-Hub/adapter-transformers}} and continued pretraining all the weights in adapter layers on target text corpus while keeping the original parameters in RoBERTa fixed. After finishing the second phase of pretraining, we performed fine-tuning of RoBERTa by training the weights in the adapters and the final classification layers while keeping all of the parameters in RoBERTa frozen. 

\subsection{Dataset}
Following \citealp{gururangan-etal-2020-dont} \footnote{Downloadble link for task dataset: \url{https://github.com/allenai/dont-stop-pretraining}}, we consider 8 classification tasks from 4 different domains. The specification of each task is shown in Table \ref{tab:data}. We covered news and review texts that are similar to the pretraining corpus of RoBERTa as well as scientific domains in which text corpora can have largely different distributions from those of RoBERTa. Furthermore, the pretraining corpora of the target tasks include both large and small cases to determine whether the adapter-based approach can be applicable in both low and high-resource settings. 

\subsection{Implementation Details}
Our implementation is based on HuggingFace since we found AllenNLP \citep{gardner-etal-2018-allennlp} used in \citealp{gururangan-etal-2020-dont} is incompatible with adapter-transformer \citep{pfeiffer-etal-2020-adapterhub}. We follow the hyperparameters setting in \citealp{gururangan-etal-2020-dont}, and each model in the pretraining and fine-tuning stage is trained on a single GPU (NVIDIA RTX 3090). Details of hyperparameters are described in Appendix \ref{sec:hyper_datails}. Note that for the pretraining step, we use a batch size of 8 and accumulate the gradient for every 32 steps to be consistent with the hyperparameter setting in \citealp{gururangan-etal-2020-dont}. 

We perform pretraining with the self-supervised objectives, which are randomly masked tokens, with a probability of 15\% for each epoch and we do not apply validation to pretraining and save the model at the end of the training from a single seed. For TAPT, we train the entire parameters of the RoBERTa via masked language modeling (MLM) on the target dataset, whereas for the adapter-based model, we embed the language adapters in each transformer layer and add invertible adapters after the embedding layers to perform MLM while freezing the original parameters of RoBERTa, following \citealp{pfeiffer-etal-2020-mad}. Fine-tuning step is straightforward. We perform fine-tuning parameters that are pretrained via MLM for both TAPT and the adapter model. Validation is performed after each epoch and the best checkpoint is loaded at the end of the training to evaluate the performance on the test set.

\begin{table*}
\begin{center}
\small
\begin{tabular}{|l|c|c|c|c|}
\hline
             
Dataset       & Baseline RoBERTa & TAPT     & Adapter w/o PT & Adapter w/ PT   \\
\hline \hline
\textsc{CHEMPROT}      & 81.9 $_{1.0}$            &  82.6 $_{0.4}$  & 82.69  $_{0.4}$& \textbf{82.71}  $_{0.4}$\\
\textsc{RCT}           & 87.2 $_{0.1}$            & \textbf{87.7} $_{0.1}$   & 87.35  $_{0.04}$& 87.4 $_{0.1}$\\
\textsc{ACL-ARC}       & 63.0 $_{5.8}$            & 67.4 $_{1.8}$ & \textbf{69.47}  $_{2.4}$& 69.25 $_{2.5}$\\
\textsc{SCIERC}        & 77.3 $_{1.9}$            & 79.3 $_{1.5}$ & 81.5  $_{0.9}$& \textbf{82.37} $_{1.0}$\\
\textsc{HYPERPARTISAN} & 86.6 $_{0.9}$            & 90.4 $_{5.2}$ & \textbf{93.01}  $_{4.7}$& 84.97 $_{6.4}$\\
\textsc{AGNEWS}       & 93.9 $_{0.2}$            & \textbf{94.5} $_{0.1}$ & 94.00  $_{0.1}$& 93.94 $_{0.1}$\\
\textsc{HELPFULNESS}   & 65.1 $_{3.4}$             & 68.5 $_{1.9}$ & \textbf{70.96}  $_{0.6}$& 70.83 $_{0.8}$\\
\textsc{IMDB}          & 95.0 $_{0.2}$            & 95.5 $_{0.1}$ & 95.51  $_{0.1}$& \textbf{95.57} $_{0.1}$\\
\hline
Average $F_1$    & 81.3             & 83.24 & \textbf{84.31}       & 83.38 \\
\hline
\hline
Trainable params per task (PT/FT) & -/124.64M     & 163.35M/124.64M & -/\textbf{1.78M}        & 2.18M/2.08M \\
Ratio to total params (PT/FT) & -/100\%     & 100\% /100\%  & -/\textbf{1.42\%}       & 1.32\%/1.65\% \\
Relative training speed (PT/FT) & -/1.0 & 1.0/1.0 & -/\textbf{1.29} & 1.14/1.24 \\ 
Relative inference speed (PT/FT) & -/\textbf{1.0} & \textbf{1.0}/\textbf{1.0} & -/0.98 & 0.88/0.98 \\ 
\hline
\end{tabular}
\end{center}
\caption{Average $F_1$ score with standard deviation on test set. Each score is averaged over 5 random seeds. Evaluation metric is macro-$F_1$ scores on test set for each task except for \textsc{CHMEPROT} and \textsc{RCT} which use micro-$F_1$. We report the results of baseline RoBERTa and TAPT from \citealp{gururangan-etal-2020-dont}. Following \citealp{ruckle2020adapterdrop}, we measure the average relative speed for the training and the inference time across all tasks except for the the inference speed in fine-tuning stage, which excludes low-resource tasks. PT and FT indicate pretraining and fine-tuning respectively.}
\label{tab:f10test}
\end{table*}

\subsection{Experimental setup}
Experiments cover four different models. First, we reproduce the performance of RoBERTa and TAPT in \citealp{gururangan-etal-2020-dont} as presented in Appendix \ref{sec:replication}. Then we proceed to the adapter-based approach. To investigate the benefits of task-adaptive pretraining with adapters, we compare the performance of the pre-trained adapter model with the model without pretraining, i.e., directly fine-tuning adapters in RoBERTa on the target task.

For the adapter-based approach, we compare the adapter-based model with the second phase of pretraining and the model without the pretraining. Since the weights of the adapters are randomly initialized, we empirically found that a larger learning rate worked well compared to the full fine-tuning experiments. We sweep the learning rates in \{2e-5, 1e-4, 3e-4, 6e-4\} and the number of epochs in \{10, 20\} on the validation set and report the test score that performs the best on the validation set.

\subsection{Results}

The results are summarized in Table \ref{tab:f10test}. Surprisingly, for the average $F_1$ score, the adapter-based model without task-adaptive pretraining performs best, followed by the other adapter with the pretraining model, TAPT, and the baseline RoBERTa. Except for Hyperpartisan news, the adapter model without pretraining performs mostly on par with the counterpart adapter model that involves pretraining on target text corpus, suggesting that the benefits of additional task-adaptive pretraining diminish when we use the adapter-based approach. Furthermore, directly fine-tuned adapter model only trains 1.42\% of the entire parameters which leads to the 30\% faster-training step than the full model and skips the pretraining stage that typically expensive to train than the fine-tuning, substantially reducing the training time while the relative speed for the inference only decreases by 2\% to the full model.



\begin{figure} 
\centering
\includegraphics[width=0.85\linewidth]{lr_f12.png}
\caption{$F_1$ score as a function of learning rate on test set with log scale on x-axis. $F_1$ score is averaged over 5 random seeds for low-resource tasks (\textsc{CHEMPROT, ACL-ARC, SCIERC, HYPER}) due to the high variance. For high-resource tasks (\textsc{RCT, AGNEWS, HELPFULNESS, IMDB}), we report the $F_1$ score from a single random seed for each task. For RoBERTa and TAPT, we follow the hyper-parameter settings in \citealp{gururangan-etal-2020-dont} except for the learning rate.}
\label{fig:f1_lr_test}
\end{figure}

\subsection{Analysis}
We analyze how the adapter alone can surpass or perform on par with both the full model and adapter model with task-adaptive pretraining. Since we sweep the learning rates and the number of epochs in the range that includes larger figures compared to those in the full model when fine-tuning adapters and kept the other hyper-parameters the same as in \citealp{gururangan-etal-2020-dont}, we hypothesize that the larger learning rate zeroes out the benefits of pretraining. Figure \ref{fig:f1_lr_test}. shows the average $F_1$ score across all tasks as a function of learning rate. 

The adapter model without a second phase of pretraining consistently outperforms or performs on par with the adapter model with pretraining from 1e-4 to 6e-4, demonstrating that the additional pretraining turns out to be ineffective. In contrast, TAPT outperforms baseline RoBERTa from 2e-5, where both TAPT and baseline RoBERTa perform best. The results show that different learning rates used in the fine-tuning stage can affect the effectiveness of pretraining and demonstrate that directly fine-tuning a fraction of parameters can provide comparable performance to the full-model as well as the adapter model with pretraining while substantially reducing the training time.


\begin{table}
\begin{center}
\small
\begin{tabular}{|l|c|c|c|c|}
\hline 
Dataset & Baseline RoBERTa  & TAPT\\
\hline \hline
\textsc{CHEMPROT}      &  \textbf{82.8} $_{0.9}$      &  82.62 $_{0.5}$ \\
\textsc{RCT}           & 86.89 $_{0.1}$           &  \textbf{87.4} $_{0.2}$\\
\textsc{ACL-ARC}       & 69.24 $_{2.6}$           & \textbf{70.08} $_{2.3}$\\
\textsc{SCIERC}        & 80.59 $_{0.9}$          & \textbf{81.28} $_{1.2}$ \\
\textsc{HYPER}         & \textbf{94.53} $_{2.0}$           & 86.17$_{1.3}$\\
\textsc{AGNEWS}        & 93.9 $_{0.2}$           & \textbf{94.05}$_{0.1}$\\
\textsc{HELPFUL}      & 69.63 $_{0.6}$           & \textbf{71.28}$_{0.8}$\\
\textsc{IMDB}          & 94.93 $_{0.1}$            & \textbf{95.33}$_{0.1}$\\
\hline
Average $F_1$          & \textbf{84.06}            & 83.52\\
\hline
\end{tabular}
\end{center}
\caption{Best performance of baseline RoBERTa and TAPT \citep{gururangan-etal-2020-dont} on our implementation. Each score is averaged over 5 random seeds. Best configuration settings for each task is described in Appendix Table \ref{tab:addex_val}.}
\label{tab:addex}
\end{table}


Inspired by the results of the adapter models, we perform the same experiments for the full model (baseline RoBERTa and TAPT) on our implementation by sweeping the learning rates and the number of epochs. We hypothesize that proper hyper-parameter settings such as a larger learning rate or increasing the number of training steps in the fine-tuning stage can improve the performance of baseline RoBERTa, making pretraining on the unlabeled target task less effective. We sweep the learning rate in \{1e-5, 2e-5, 3e-5\} and the number of epochs in \{10, 20\} on the validation set and report the test score that performs the best on the validation set. Table \ref{tab:addex} shows the best performance of the full models for each task among different hyper-parameter settings. The average  $F_1$ score of baseline RoBERTa greatly increases and surprisingly, it surpasses the performance of TAPT in some tasks. The results ensure that although pretraining PLMs on the target task results in better performance, one can achieve comparable performance by simply using a larger learning rate or increasing training steps in the fine-tuning stage while skipping the pretraining step that is computationally demanding compared to the fine-tuning.


\section{Conclusion}
\label{sec:conclude}
Our work demonstrates that adapters provide a competitive alternative to large-scale task-adaptive pretraining for NLP classification tasks.  We show that it is possible to achieve similar performance to TAPT with pretraining training just 1.32\% of the parameters through pretraining with adapters. However, the most computationally efficient option is to skip pretraining and only perform fine-tuning with adapters. We found that skipping pretraining altogether and just fine-tuning with adapters outperforms or performs mostly on par with TAPT and the adapter model with pretraining across our tasks while substantially reducing the training time.

\section*{Acknowledgments} 
We would like to thank Zsolt Kira, Mandeep Baines, Shruti Bhosale, and Siddharth Goyal for helpful feedback and suggestions. We also would like to thank anonymous reviewers for their insightful comments on the earlier version of the paper.

\bibliographystyle{acl_natbib}
\bibliography{anthology,acl2021}

%\clearpage
\appendix
%\section*{Appendix}

\section{Hyperparameter Details}
\label{sec:hyper_datails}
Details of hyperparameter setting including the learning rates for the best performing results are provided in Table \ref{tab:hyper_pt}, \ref{tab:hyper_ft}, and \ref{tab:hyperlr}.

\section{Validation Results}
\label{valid_results}
We present validation performance in Table \ref{tab:f10val} and Figure \ref{fig:f1_lr_dev}.

\section{Replication results}
We provide replication results of \citealp{gururangan-etal-2020-dont} in Table \ref{tab:repro}.
\label{sec:replication}

\begin{table*}
\begin{center}
\begin{tabular}{l|c} 
\textbf{Hyper-parameter} & \textbf{Value}\\
\hline
Optimizer & Adam \\
Adam epsilon & 1e-8, 0.999 \\
Learning rate & 1e-4 \\ 
Batch size & 8 \\ 
Gradient accumulation step & 32 \\ 
Epochs & 40 or 100 \\ 
Adapter reduction factor & 12 \\ 
Maximum sequence length & 512 \\
\hline
\end{tabular}
\end{center}
\caption{Details of hyperparameters used in pretraining experiments. We used 40 number of epochs for \textsc{HELPFULNESS} and 100 for the other tasks.} 
\label{tab:hyper_pt} 
\end{table*}

\begin{table*}
\begin{center}
\begin{tabular}{l|c} 
\textbf{Hyper-parameter} & \textbf{Value}\\
\hline
Optimizer & Adam \\
Adam epsilon & 1e-8, 0.999 \\
Batch size & 16 \\ 
Gradient accumulation step & 1 \\ 
Epochs & 10 or 20 \\ 
Patience & 3 or 5 \\ 
Adapter reduction factor & 12 \\ 
Dropout  & 0.1 \\ 
Feedforward layer & 1 \\
Feedforward nonlinearity & tanh \\
Classification layer & 1 \\
Learning rate & see Table \ref{tab:hyperlr} \\
Learning rate decay & linear \\
Warmup proportion & 0.06 \\
Maximum sequence length & 512 \\
\hline
\end{tabular}
\end{center}
\caption{Details of hyperparameters used in fine-tuning experiments. For baseline RoBERTa and TAPT, we used 10 number of epochs with patience of 3 and the learning rate of 2e-5. For adapter experiments, see Table \ref{tab:hyperlr}.} 
\label{tab:hyper_ft} 
\end{table*}


\begin{table*}
\begin{center}
\small
\begin{tabular}{|l|c|c|}
\hline
             
Dataset &  Adapter w/o PT (LR, Epochs, Patience) & Adapter w/ PT (LR, Epochs, Patience)  \\
\hline \hline
\textsc{CHEMPROT}      & 3e-4, 20, 5          & 6e-4, 20, 5  \\
\textsc{RCT}           & 1e-4, 10, 3          & 1e-4, 10, 3  \\
\textsc{ACL-ARC}       & 6e-4, 10, 3          & 6e-4, 20, 5 \\
\textsc{SCIERC}        & 3e-4, 20, 5          & 6e-4, 20, 5  \\
\textsc{HYPER}         & 3e-4, 20, 5          & 1e-4, 20, 5   \\
\textsc{AGNEWS}        & 1e-4, 10, 3          & 1e-4, 10, 3   \\
\textsc{HELPFUL}       & 3e-4, 20, 5          & 1e-4, 20, 5  \\
\textsc{IMDB}          & 1e-4, 10, 3          & 1e-4, 10, 3   \\
\hline
\end{tabular}
\end{center}
\caption{Learning rate, the nubmer of epochs and patience for best-performing models. For adapter experiments, we sweep the learning rates in  \{1e-4, 3e-4, 6e-4\}, the number of epochs in \{10, 20\}, and patience factor in \{3, 5\} on validation set.}
\label{tab:hyperlr}
\end{table*}

%\clearpage

\begin{table*}
\begin{center}
\small
\begin{tabular}{|l|c|c|}
\hline
             
Dataset & Adapter w/o pretraining & Adapter w/ pretraining \\
\hline \hline
\textsc{CHEMPROT}      & 83.77 $_{0.5}$    &84.02 $_{0.7}$ \\
\textsc{RCT}            & 88.16 $_{0.1}$   &88.13 $_{0.1}$\\
\textsc{ACL-ARC}       & 72.41 $_{2.2}$   & 77.31 $_{2.9}$\\
\textsc{SCIERC}         & 86.86 $_{0.5}$ & 87.87 $_{0.3}$ \\
\textsc{HYPER}        & 86.33 $_{1.4}$    & 86.00 $_{3.5}$\\
\textsc{AGNEWS}        & 94.28 $_{0.1}$   & 94.57 $_{0.1}$\\
\textsc{HELPFUL}                 & 70.83 $_{1.2}$   & 70.8 $_{0.7}$\\
\textsc{IMDB}                & 95.52 $_{0.1}$   & 95.6 $_{0.1}$\\
\hline
Average $F_1$     & 84.77   & 85.54 \\
\hline

\end{tabular}
\end{center}
\caption{Validation performance of adapter experiments. Each score is averaged over 5 random seeds. Evaluation metric is macro-$F_1$ scores for each task except for \textsc{CHMEPROT} and \textsc{RCT} which use micro-$F_1$.}
\label{tab:f10val}
\end{table*}


\begin{figure*} 
\centering
\includegraphics[width=0.5\linewidth]{lr_f12_val.png}
\caption{$F_1$ score as a function of learning rate on development setwith log scale on x-axis. $F_1$ score is averaged over 5 random seeds for low-resource tasks (\textsc{CHEMPROT, ACL-ARC, SCIERC, HYPER}) due to the high variance. For high-resource tasks (\textsc{RCT, AGNEWS, HELPFULNESS, IMDB}), we report the $F_1$ score from a single random seed for each task. Here we sweep the learning rate in \{1e-4, 3e-4, 6e-4\}, the number of epochs in \{10, 20\}, and the patience factor in \{3, 5\}.}
\label{fig:f1_lr_dev}
\end{figure*}


\begin{table*}
\begin{center}
\small
\begin{tabular}{|l|c|c|c|c|c|}
\hline 
Dataset & Baseline RoBERTa  & TAPT & Hyper-parameters (LR, Epochs, Patience)\\
\hline \hline
\textsc{CHEMPROT}      &  \textbf{82.8} $_{0.9}$      &  82.62 $_{0.5}$ & 3e-5, 20, 5 \\
\textsc{RCT}           & 86.89 $_{0.1}$           &  \textbf{87.4} $_{0.2}$ & 2e-5, 10, 3\\
\textsc{ACL-ARC}       & 69.24 $_{2.6}$           & \textbf{70.08} $_{2.3}$ & 3e-5, 20, 5\\
\textsc{SCIERC}        & 80.59 $_{0.9}$          & \textbf{81.28} $_{1.2}$ & 2e-5, 20, 5\\
\textsc{HYPER}         & \textbf{94.53} $_{2.0}$           & 86.17$_{1.3}$ & 3e-5, 10, 3\\
\textsc{AGNEWS}        & 93.9 $_{0.2}$           & \textbf{94.05}$_{0.1}$ & 2e-5, 10, 3\\
\textsc{HELPFUL}      & 69.63 $_{0.6}$           & \textbf{71.28}$_{0.8}$ & 2e-5, 20, 5\\
\textsc{IMDB}          & 94.93 $_{0.1}$            & \textbf{95.33}$_{0.1}$ & 2e-5, 20, 5\\
\hline
Average $F_1$          & \textbf{84.06}            & 83.52 & \\
\hline
\end{tabular}
\end{center}
\caption{Validation performance of Baseline RoBERTa and TAPT experiments that corresponds to Table \ref{tab:addex}. Each score is averaged over 5 random seeds. }
\label{tab:addex_val}
\end{table*}


\clearpage

\begin{table*}
\begin{center}
\small
\begin{tabular}{|l|l|l|l|l|}
\hline
              & Original Results & Original Results & Our Results & Our Results\\
Dataset       & Baseline RoBERTa & TAPT     & Baseline RoBERTa & TAPT     \\
\hline \hline
CHEMPROT      & 81.9 $_{1.0}$             & 82.6  $_{0.4}$ & 81.64  $_{0.8}$        & 82.58 $_{0.5}$\\
RCT           & 87.2 $_{0.1}$           & 87.7 $_{0.1}$ & 86.89 $_{0.1}$         & 87.4 $_{0.2}$\\
ACL-ARC       & 63.0 $_{5.8}$              & 67.4 $_{1.8}$ & 64.12 $_{5.5}$                 & 66.11 $_{4.6}$        \\
SCIERC        & 77.3  $_{1.9}$            & 79.3 $_{1.5}$ & 78.89 $_{2.7}$                 & 79.94 $_{0.7}$        \\
HYPER         & 86.6  $_{0.9}$             & 90.4 $_{5.2}$ & 85.03 $_{6.0}$                 & 91.56 $_{2.5}$        \\
AGNEWS        & 93.9  $_{0.2}$            & 94.5  $_{0.1}$ & 93.72 $_{0.2}$                 & 94.05 $_{0.1}$        \\
HELPFULNESS   & 65.1   $_{3.4}$           & 68.5 $_{1.9}$  & 69.2   $_{1.4}$                & 71.24 $_{0.7}$        \\
IMDB          & 95.0   $_{0.2}$           & 95.5  $_{0.1}$ & 95.15  $_{0.1}$                & 95.33 $_{0.1}$        \\
\hline
Average $F_1$    & 81.3             & 83.24 & 81.83          & 83.53 \\
\hline
\end{tabular}
\end{center}
\caption{Reproducing Baseline RoBERTa and TAPT Results, average $F_1$ Scores with standard deviation. $F_1$ score is averaged over 5 random seeds. We use the same hyper-parameters in \citealp{gururangan-etal-2020-dont}.}
\label{tab:repro}
\end{table*}

%\clearpage


\end{document}


