%
% File acl2021.tex
%
%% Based on the style files for EMNLP 2020, which were
%% Based on the style files for ACL 2020, which were
%% Based on the style files for ACL 2018, NAACL 2018/19, which were
%% Based on the style files for ACL-2015, with some improvements
%%  taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by 
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith

\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{acl2021}
\usepackage{times}
\usepackage{latexsym}
\renewcommand{\UrlFont}{\ttfamily\small}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}


\usepackage{xspace}
\usepackage{booktabs}
\usepackage{multicol}
\usepackage{amsmath}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{mdwlist}
\usepackage{xcolor}

\newcommand{\daj}[1]{\textcolor{red}{\bf\small [#1 --DJ]}}

\newcommand{\hd}[1]{\textcolor{blue}{\bf\small [#1 --HD]}}

\newcommand{\ptft}{\textsc{PtFt}\xspace}
\newcommand{\ft}{\textsc{Ft}\xspace}

\newcommand\bn[0]{\textcolor{orange}{\textsc{Number}}\xspace}
\newcommand{\nc}{\textcolor{purple}{\textsc{Citation}}\xspace}

\newcommand{\Sref}[1]{\S\ref{#1}}
\newcommand{\sref}[1]{\S\ref{#1}}
\newcommand{\fref}[1]{Figure~\ref{#1}}
\newcommand{\tref}[1]{Table~\ref{#1}}

%\newcommand{\myparagraph}[1]{\noindent \textbf{#1}}
\newcommand{\myparagraph}[1]{\paragraph{#1}}

\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{***} %  Enter the acl Paper ID here

%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.

\newcommand\BibTeX{B\textsc{ib}\TeX}

\title{HamiltonDinggg at SemEval-2021 Task 5: Investigating Toxic Span Detection using RoBERTa Pre-training}

\author{Huiyang Ding \\
  School of Information \\
  University of Michigan  \\
  Ann Arbor, MI 48105 \\
  \texttt{huiyangd@umich.edu} \\\And
  David Jurgens \\
  School of Information \\
  University of Michigan  \\
  Ann Arbor, MI 48105 \\
  \texttt{jurgens@umich.edu} \\}

\date{}

\begin{document}
\maketitle
\begin{abstract}
This paper presents our system submission to task 5: Toxic Spans Detection of the SemEval-2021 competition. The competition aims at detecting the spans that make a toxic span toxic. In this paper, we demonstrate our system for detecting toxic spans, which includes expanding the toxic training set with Local Interpretable Model-Agnostic Explanations (LIME), fine-tuning RoBERTa model for detection, and error analysis. We found that feeding the model with an expanded training set using Reddit comments of polarized-toxicity and labeling with LIME on top of logistic regression classification could help RoBERTa more accurately learn to recognize toxic spans. We achieved a span-level F1 score of 0.6715 on the testing phase. Our quantitative and qualitative results show that the predictions from our system could be a good supplement to the gold training set's annotations.
\end{abstract}

%  We note that toxicity itself can be challenging to measure \cite{vidgen2019challenges}. Multiple models have been proposed for handling different aspects of antisocial behavior  \cite{fortuna2018survey} and that these models frequently have gaps in what they recognize \cite{arango2019hate,jurgens2019just}, can encode biases with respect to social groups \cite{sap2019risk}, and be susceptible to adversarial attacks \cite{grondahl2018all}. Nevertheless, as a widely-used measure, Perspective API provides a replicable---though imperfect---tool specifically designed for recognizing multiple forms of toxicity in online conversations in comments, which is the medium studied here.

\section{Introduction}

Toxic messages remain a small but persistent part of online communications \cite{fortuna2018survey,jurgens2019just}. NLP methods have been developed to identify these comments, often relying on deep-language models \cite{vidgen2019challenges}. However, the part of the message that is specifically toxic is often unknown. Such information is useful not only for validating and explaining the judgments of models \cite{carton2018extractive}, but can also be useful for moderators to use when making decisions and working with these models in their deployment \cite{carton2020feature,liu2021understanding}. This paper describes our model\footnote{The code is available at \url{https://github.com/davidjurgens/offensive-span-detection}.} and error analysis for SemEval-2021 Task 5: Toxic Spans Detection \cite{pav2020semeval}. All the data mentioned in the paper is in English language primarily.

Our model uses a deep learning approach to identify which tokens are toxic. The approach is motivated by two strands of prior work showing (1)  that large language models can effectively serve as sequence-to-sequence (seq2seq) models and (2) that pre-training on a similar task can improve downstream performance \cite{phang2018sentence,gururangan2020don}. Here, we treat the toxic-span detection tasks as a seq2seq task, where given a sequence of tokens, the model outputs per-token judgments of whether the token is in the toxic span. Given the limited training data for Task 5, we increase our training data by generating a silver-standard set of span judgments from LIME explanations \cite{ribeiro2016should} fropm a model trained to recognize toxic and non-toxic language. These additional judgments are intended to help the model learn the basic span recognition task and identify general toxic language, before fine-tuning on the Task 5 data.



%The Task5 in SemEval 2021 is a \textbf{sentiment analysis} tasks that requires the positioning of toxic spans in a toxic comment\footnote{\url{https://competitions.codalab.org/competitions/25623}}. It requires systems that could take in a document as input and output a list of indices of the detected toxic spans. The dataset along with the annotations is created based on Civil Comments\footnote{\url{https://www.tensorflow.org/datasets/catalog/civil_comments}} dataset's toxic English comments on news sites.

%As the NLP community is focusing more on unveiling the deep models' "black box" by leveraging transfer learning, the task of highlighting spans that drive model decisions is a cut-edging area. This paper seeks for a pre-training method to improve the performance of RoBERTa model(\citet{liu2019roberta}) on this kind of task.




\section{System Description}

Our core system relies on a standard RoBERTa model \cite{liu2019roberta} that is trained on a sequence-to-sequence task in two phases. The first phase pretrains the model with heuristically-created spans, gathered from Reddit comments labeled for their toxicity. The second phase fine-tunes this model on the organizer-provided data. Figure~\ref{fig:system-diagram} shows the overview of the system.


%\footnote{The data was collected by Blablablab. The details could be found in the lab website: \url{https://blablablab.si.umich.edu/projects/support/}}; the second phase involves in the fine-tuning of the pre-trained RoBERTa model on the original(gold) dataset.

\begin{figure}[tb]
    \centering
    \includegraphics[scale=0.75]{Final.pdf} %[width=0.9\linewidth, 
    \caption{Diagram of our data and architecture. The central hypothesis tested is whether pre-training a RoBERTa model on machine-generated rationales for toxicity could improve performance.}
    \label{fig:system-diagram}
\end{figure}


\subsection{Pretraining to Recognize Toxicity}

To identify toxic spans, we hypothesize that pre-training the RoBERTa model on a similar task would lead to better downstream performance. Therefore we generate a similar dataset (silver dataset) to the training data (gold dataset) and heuristically label it with spans by using LIME \cite{ribeiro2016should} on a toxicity classification task.

\paragraph{Data}

%According to the RoBERTa authors' implementation\footnote{\url{https://github.com/pytorch/fairseq/tree/master/examples/roberta}}, RoBERTa-base model has a BERT-base architecture with 12 layers and 768 hidden units, which adds up to 125 million parameters. Thus, a much bigger training set compared to the gold dataset would benefit RoBERTa's training phase. Thus, Reddit data from Blablablab was used to expand the training set. 

Silver data was drawn from a sample of all Reddit comments made between January to June 2018. As social media data, these comments contain similar lexical and syntactic patterns as the social media comments data as the gold standard, which was made on the Civil Comments platform. Prior work has shown that pre-training RoBERTa models to recognize this type of social media data improve downstream performance \cite{nguyen-etal-2020-bertweet}. However, Reddit posts can vary substantially in their length. To avoid introducing confounding effects from pre-training a model on posts of substantially different lengths, we compute the Inter-quartile Range (IQR) of the lengths of Reddit comments and remove all comments identified as outliers. This process effectively removes very long or very short comments. Ultimately, the mean number of words in the training data and Reddit data are roughly similar: 35.87$\pm$34.92 words (mean and standard deviation) in the training data, compared with  36.79$\pm$30.57 in the Reddit data.

\paragraph{Identifying Toxic Comments}

The Reddit data contains a mix of toxic and non-toxic conversations, which we aim to use for training. To identify toxic conversations, we use the Perspective API to label all comments in the dataset \cite{wulczyn2017ex}. The API returns a continuous score reflecting the degree of a comment's toxicity. This toxicity score is then converted into a binary label to use in training a LIME model to generate rationales for why a comment is (or is not) toxic. We follow the insights from \citet{hua2020towards} and set a threshold of 0.7, above which a comment is considered toxic and 0.3, below which the comment is non-toxic. These thresholds were intended to help create easy examples of toxic language for generating rationales as a way of scaffolding the learning for the downstream task.
%
This process led to a labeled dataset of 288.5M comments with binary toxicity labels, of which 9.4\% were labeled toxic.

% 288577570

% Based on human-reviewing, Reddit data used range from January, February, March, May, June in year 2018 to resemble the topics of the gold dataset. Through randomized sampling on top of the documents within the above-mentioned thresholds, datasets with different toxic-nontoxic ratio were generated. The datasets were built with 400000 toxic documents and various size of nontoxic documents. For the experimental setup, we added 0, 200000, 400000 nontoxic documents onto the 400000 toxic documents and composed 3 silver datasets.

\paragraph{Generating Heuristically-Labeled Toxic Spans}

Our final silver dataset is created by sampling comments from the larger labeled Reddit comments and using a LIME model to generate toxic span labels.  LIME is a form of interpretable machine learning that explains the decisions of a classifier using a local approximation to identify which features led to a classification decision. Here, we use a simple logistic regression (LR) model trained on TF-IDF features and use LIME to generate a rationale of the classifier's decision which identifies which words are contributing to the toxicity decision.  
%
The underlying LR model is trained on a balanced sample of 800K toxic/non-toxic comments (not the silver data). This balanced sample is derived in the same way as silver data. The model's hyperparameters were tuned using 10-fold cross-validation, with thae learning rate of 0.01 and strength of the regularization (C) at 1 under L1 loss. 
%
In a test of a held-out 200K instances, the model attained a binary F1 of 0.985. % Note that while this performance is quite high, the data's labels are designed to be easily distinguishable 

%According to the paper of LIME (), LIME aims at using the local-aware loss minimization and sparse linear explanations to achieve interpretabilities on tasks such as text classification. Mathematically, LIME is trying to minimize the loss with the locally weighted square loss function, such that:

%$\mathcal{L}\left(f, g, \pi_{x}\right)=\sum_{z, z^{\prime} \in \mathcal{Z}} \pi_{x}(z)\left(f(z)-g\left(z^{\prime}\right)\right)^{2}$

%where f is the probability that x (tokenized document) belongs to a class (toxic/non-toxic). G is a potentially interpretable linear model that $g\left(z^{\prime}\right)=w_{g} \cdot z^{\prime}$ with a domain of $\{0,1\}^{d^{\prime}}$, which are used over presence/absence of components (tokens), and $\pi_{x}(z)=$ $\exp \left(-CosineDis(x, z)^{2} / \sigma^{2}\right)$ is an exponential kernel defined on cosine distance. 

Our silver data is created by generating LIME explanations using the trained LR classifier on a separate 800K comments balanced between toxic and non-toxic. This size is roughly 100x the Task-provided data. In generating explanations, LIME assigns local weights to each token on its weight to drive the correct prediction. To create toxic spans from these continuous-valued weights, we apply a threshold above which we consider the token as the toxic span. The threshold was identified by generating LIME explanations for all of 8629 documents from the Task's training data and then choosing the threshold that maximized the Span F1 between the 8629 training documents' toxic spans and the discretized LIME explanations, using a grid search with a step of 0.001 in [0.05,0.50]; the final threshold was set to 0.169.

%On top of the classification pipeline, LIME is applied to label the locally-significant tokens that were driving the classifier's prediction with the contributions of tokens. LIME takes the permutations of tokenized embedding results for each instance of the document, thus creates simpler models using these reduced-size input. In this way, the weights of tokens in each simpler model instance are re-evaluated according to similarity between the ground-truth prediction and the new model prediction. Repeatedly applying the local surrogate algorithm, LIME would


\subsection{Model Training}

Our model uses a common RoBERTa base and differs according to which data the model is trained on.
%
%Our model users both the Task-provided data and our heuristically-labeled silver data to train a RoBERTa model. We refer to training on the task data as Fine Tuning and training on th
%
%The silver sets are used to train the RoBERTa-base models, with both toxic and non-toxic documents included. The training set was trained in 2 epochs and evaluated on the gold dataset. For the validation purpose, this pre-trained model is evaluated on the trail dataset and tested on the test dataset released by the organizer.
%
%\paragraph{Pre-Training}
%
The pre-training setup trains a RoBERTa model on a seq2seq task where the input sequence of tokens generates a binary sequence denoting whether the input token was inside or outside of a toxic span. Due to the dataset sizes, pre-training was done for one epoch.
%
%\paragraph{Fine-tuning}
%
The fine-tuning setup starts from either the off-the-shelf RoBERTa parameters or from a RoBERTa model initialized through silver-data pre-training. This model is trained in the same way as in pre-training on a binary seq2eq task using the Task-provided data. Models are fine-tuned for 10 epochs and parameters are chosen using the epoch with the best performance on the trial data.
%

In internal testing, we compared models that have been pre-trained, fine-tuned, or both, using varying amounts of silver data. All hyperparameter choices are reported in the Appendix \ref{appendix:a}.

%In this phase, the pre-trained RoBERTa models from phase 1 are fine-tuned on the gold dataset, using the locked parameters with best performing models yielding the best evaluation f-1 scores. Each document is firstly tokenized by RoBERTa tokenization. Then the tokenized document is batched and fed into RoBERTa model with a linear token binary-classification layer on top of the hidden-states output. The tokens are masked as toxic and non-toxic, and the classification layer predicts the toxicity for each token. The results are again evaluated on the trail dataset released by the organizer and tested on the test dataset using the span-level f-1 score. Each of the fine-tuning is performed with 10 epochs training. The parameters are locked for testing on the best evaluation f-1 scores.

% As for baselines, the pre-trained RoBERTa-base model was trained on gold dataset only with 10 epochs, without involving our system's pre-training phase. For comparison purpose, we also pre-trained RoBERTa on full silver set in one epoch and directly tested on test dataset.

% Moreover, in order to capture the toxic spans that were miss-spelled and textspeak, we introduced a rule-based labeling strategies using the work from XXX. In employing the offensive corpus from the work and added regular expression based expansion on the corpus, it consists of a toxic vocabulary. Any span, word, or phrase that matches the instances in the toxic vocabulary will labeled as toxic, which could supplement the result of modeling detection.


\section{Results}

Our best model attained a Span F1 of 0.672, and although close in score to the top result (0.708), was ranked 30 in the Task. Surprisingly, this best-performing model did \textit{not} make use of the pre-training on silver data. To better understand the performance, we ran two follow-up analyses to test how different strategies for training affected performances and an error analysis for what were common themes in errors. 



\begin{table}[t]
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{c c cc }

\textbf{Model} & \textbf{Batch Size} & \textbf{Silver Data} &  \textbf{ F1} \\ 
\hline
FT                &  8 & N/A               & \textbf{0.675} \\ 
FT                & 16 & N/A               & ~0.672$^{\dagger}$ \\ 
PT                &  8 & 400k/400k (1:1)   & 0.613 \\ 
PT + FT           & 8 & 400k/0~~~~~~ (1:0)      & 0.660 \\
PT + FT           & 8 & 400k/200k (2:1)    & 0.660 \\
PT + FT           & 8 & 400k/400k (1:1)    & 0.659 \\ 
\end{tabular}
}
\caption{Performance at recognizing toxic spans (Span F1) for models trained on just the Task-provided training data (baseline), Pre-Trained (PT) on different amounts and ratios of silver data, and Fine-Tuned (FT) on training data. Ratios denote the number of non-toxic:toxic examples. ~$^\dagger$ is the model submitted to the Task. }
\label{tab:performance}
\end{table}

%The result show that our system models do not perform better quantitatively better than the \ft models, but they do yield close span f-1 scores. However, if we skip the fine-tuning section and only use the model from pre-training phase, the performance is significantly lower. Different ratios of toxic and non-toxic documents in the silver set did not impact much on the final testing performances. Given that the \ft models and the models from our system having close quantitative performance, it becomes interesting to see what these models do similarly and differently.


%\subsection{Error Analysis}

%Trough analyzing the errors, we found it valuable to present (1) the similarities and differences between the results from \ft models and system models; (2) the differences between the annotators' annotation decisions and system models' output.

\subsection{Does Pre-training Make a Difference?}
\label{sec:pretraining}

In assessing the impact of pre-training, we analyze the submitted model along with five other models: (1) a fine-tuned model using a batch size of 8, (2) a pre-trained only model that makes no use of the Task data, and (3-5) pre-trained and fine-tuned models that use different amounts of silver data. The performances of all models are shown in \tref{tab:performance}.

%For comparison purposes, we are comparing the \ft model trained in batch size 8 and the \ptft model pre-trained on the full silver set (800k). We also compare their performances with the spans annotated as toxic by organizers (ground-truth spans).

For the initial comparison, we contrast the fine-tuned model (\tref{tab:performance}, Row 1; denoted \ft) with the pre-trained and fine-tuned model on all silver data (\tref{tab:performance}, Row 6; denoted \ptft).
%
Both models agree on 1281 (65\%) of the 2000 test instances. For these agreed cases, both models attain a Span F1 of 0.776---higher than either models regular performance. In these matching predictions, the ground-truth spans have an mean length of 1.13 tokens, mainly concentrated on commonly-labeled offensive words, like ``morons'',  suggesting that both models are adept at identifying overtly toxic words.  In contrast, for the 21 test documents whose spans have $\ge$5 words, both models perform poorly with a Span F1 of 0.3489 for the \ft model and 0.2077 for the \ptft model. 

The \ft model performs considerately better than \ptft model on documents with $\ge$5 tokens labeled as ground-truth spans, which is likely due to differences between the LIME-labeled data and the Task's training data. For example, the LIME model generates spans $\ge$5 tokens in only 693 of the 800K silver context, suggesting LIME tends to give shorter toxic span labels. 

This bias affects the downstream model performance in the test set where 29 of the 2000 test contexts have a span of $\ge$2 consecutive toxic words. In those contexts, the \ft model achieves a mean Span F1 of 0.523 while the \ptft model has only 0.369. Indeed, the \ft model produces spans (average span length: 19.0345) that are $\sim$224\% longer than the \ptft model (average span length: 8.4828). This difference is more obvious than the overall prediction results, as seen in \tref{table:span-lengths}.
%The average length of the spans from \ft model's output is 45.58, which is 15.37 longer than the \ptft model. In general, the \ptft model tends to predict shorter spans compared to the \ft model, as seen in \tref{table:span-lengths}.

In the remaining cases where the \ft and \ptft model predictions differ, the \ft model has 296 predictions with a better Span F1 score, of which 216 predictions have longer span. For example, in test context 100, ``Stupid is as stupid does Gump was right,'' both of the ``stupid" tokens are highlighted by the \ft model, while the \ptft model only labels out the first ``stupid". 

However, the tendency of the \ft model to predict longer spans does not always yield higher performance. The \ptft model has 237 predictions with better Span F1 scores.
%, where the model makes shorter predictions than the \ft model in 202 cases.Here, much of the performance improvement comes from recognizing non-toxic messages where no span should be reported, which the \ft model struggles with.

The \ft model has a longer span prediction in the 202 of the 237 predictions, but the qualitative results are very different from the above-mentioned example. In multiple cases when the ground-truth spans are empty, the \ptft model can also predict empty spans. However, the \ft model has a lower tendency to predict empty spans in those cases. For example, in non-toxic test context 3, ``The parallels between the ANC and the Sicilian Mafia are glaring...", the \ft model labels ``Sicilian" as toxic, while the \ptft model output is (correctly) empty. % Given a review of the context, ``Sicilian" has a neutral tone and meaning in the document, which indicates the error is made by \ft model.

Looking at contexts where there are no underlying toxic spans, the two models perform slightly differently. There are 394 out of 2000 test contexts with empty ground-truth spans. For those contexts, the \ft model only gets a mean Span F1 score of 0.058, while the \ptft model gets 0.079. 

In contrast to the \ft model, the \ptft model has less-accurate predictions on the overtly/commonly toxic spans. For example, there are 430 total ``stupid" or ``stupidity" related words labeled as toxic by the ground-truth spans. The \ft model is able to label 383/430 as toxic, while the \ptft model only labels 331/430. As we know, words like ``stupid" can be more contextually-sensitive when compared to other common offensive words. They could be used in a toxicity-neutral way in many contexts. In the \ptft model's pre-training phase, we fed 400,000 non-toxic documents for the RoBERTa model. These non-toxic documents supplied more non-offensive context for certain toxic words than the small-sized gold dataset. The extra contextual information learned by the pre-trained model can somehow decrease the performance of the \ptft model. 



\begin{table}[]
\centering
\begin{tabular}{@{}rll@{}}

\textbf{Span Size} & \textbf{\ft model} & \textbf{\ptft model} \\ \hline
Characters  & 9.30 ± 6.44    & 7.05 ± 3.48  \\
Tokens & 2.14 ± 1.72    & 1.61 ± 1.00  \\
\end{tabular}
\caption{Differences in model prediction length shows that pretraining on LIME-generated toxicity rationales (\ptft) generally produces shorter spans at both span and token levels.}
\label{table:span-lengths}
\end{table}

\subsection{Common Themes in Errors}
From the error analysis in the above section, we have noticed that the \ptft model does not perform well when it comes to predicting long toxic spans, empty toxic spans, and toxic phrases. With a deeper dive into the differences between \ptft model predictions and ground-truth spans, we can get a better sense about how is our \ptft model doing and what insights could be recommended for annotators.

In order to perform both qualitative and quantitative error analyses, we randomly sampled 200 test contexts where predictions from the \ptft model do not conform with ground-truth labels. The overall mean Span F1 score on those mispredictions is 0.289; compared to the random sample with a mean of 0.274. We categorize the mistake types from the sampled contexts and provide examples in Table \ref{tab:examples}.

\begin{table*}[!tb]
    \centering
    \begin{tabular}{l l p{0.56\linewidth}}
        
        \textbf{Category} & \textbf{\%} & \textbf{Examples} \\
        \hline
        \multirow{3}{0.3\linewidth}{1. Toxic not labeled by ground-truth spans} & \multirow{3}{*}{50.5} & 
         
         491. this guy is an \colorbox{blue!30}{idiot}. I'll disclose I'm male....  \\
        & & 1374. That is a bunch of horse \colorbox{blue!30}{sh*t}. \\ 
        & & 1776. \colorbox{blue!30}{\textbf{Stupid}} leftists. \colorbox{blue!30}{Stupid} Democrats. I sincerely.. \\
\hline

        \multirow{5}{0.3\linewidth}{2. Toxic not labeled by the \ptft model} & \multirow{5}{*}{40.5} & 
        1852. That's because Floater \colorbox{blue!30}{\textbf{sucks}}. That dudes voice is so \textbf{fucking} annoying! \\
        & & 1486. uhhh Hillary Clinton is a \colorbox{blue!30}{serial killer} and \textbf{thief}  \\
        & & 346. ...Drivers in Anchorage are lazy, \colorbox{blue!30}{\textbf{stupid}} and \textbf{impatient}... \\
        & & 642. What a load of \textbf{caca}!! ... \\
        \hline
        
        \multirow{1}{0.3\linewidth}{3. Not toxic but labeled by ground-truth spans} 
        & 0.5 &
         876. Ergo, America is overrun with the \textbf{insane}. \\
         & & \\
        \hline
        
        \multirow{2}{0.3\linewidth}{4. Not toxic but labeled by the \ptft model} & 13.5 & 
        273. Extremist \colorbox{blue!30}{Muslims} want to kill infidels. Moderate Muslims cook the extremists a hot lunch. \\
        & & 1802. either you can afford to lose \$1300. in an hour or you're a \colorbox{blue!30}{moron}
        \\
        \hline
        
        \multirow{4}{0.3\linewidth}{5. Inconsistent multiple-word/phrase annotation by ground-truth} & \multirow{4}{*}{4} & 
        % 111. boo hoo hoo. Poor ting. Mayor Caldwell is so hated. Nice try dragging Trump into this.\colorbox{yellow!30}{\textbf{Too pathetic}}.\\
        773. Very true. Still \colorbox{yellow!30}{sick \textbf{bastards}}.\\
        & & 1496. Trump is an \textbf{impulsive \colorbox{yellow!30}{idiot}}.  He will get us all killed.\\
        & & 1776. \colorbox{yellow!30}{Stupid} leftists. \colorbox{yellow!30}{\textbf{Stupid}} Democrats. I sincerely believe... \\
        & & 1447. Brooks, would you please join the \textbf{\colorbox{yellow!30}{damn} Democrat} party and be done with it ?
        \\
        \hline
        
        \multirow{5}{0.3\linewidth}{6. Inconsistent word annotation by ground-truth} & \multirow{5}{*}{41.5} & 
        968. ok then you dont use gasoline, plastic  or such anything else right??? ya \colorbox{yellow!30}{\textbf{hypocrite}} bs \textbf{stupidity} as usual   \\
        & & 348. ...Hawaii Democrats deny ordinary citizens their constitutional right to self-defense with firearms, including concealed carry... \colorbox{yellow!30}{Hypocrites}!\\
        \hline
        
        \multirow{5}{0.3\linewidth}{7. Inconsistent repeated word annotation by ground-truth} & \multirow{5}{*}{2} & 
        413. There is a difference between being tolerant and being \colorbox{yellow!30}{\textbf{stupid}}.  She and her supporters want America  to be \colorbox{yellow!30}{\textbf{stupid}}.\\
        & & 137. I cannot fathom the \colorbox{yellow!30}{\textbf{stupidity}} of McCain believing any cooperation from the ... The man must be the \colorbox{yellow!30}{stupidest} person in D.C.  
        \\
        \hline
        
        \multirow{2}{0.3\linewidth}{8. Inconsistent sentence annotation by ground-truth} & \multirow{2}{*}{12.5} & 
        894.\textbf{You are a  \colorbox{yellow!30}{nut ball.}}\\
        & & 1374.That is a bunch of  \colorbox{yellow!30}{horse sh*t}.  
        \\
        \hline
        
        % Longer Span & \bn\% & The authors  are \colorbox{blue!30}{\textbf{dumb} for not seeing how stupid they are}. \\ 
        % Longer Span & \bn\% & The authors  are \colorbox{blue!30}{\textbf{dumb} for not seeing how stupid they are}. \\ 
        % Longer Span & \bn\% & The authors  are \colorbox{blue!30}{\textbf{dumb} for not seeing how stupid they are}. \\ 
        \hline
                 
    \end{tabular}
    \caption{An error analysis of 200 predictions of our \ptft model relative to the ground-truth span. All of the sample contexts in the Examples column begin with the context index starting from 0. The first four categories show the differences in between the toxic spans labeling from \ptft model (shaded in blue) and the ground-truth span (bolded). The last four categories show the commonly-seen inconsistencies existing in the ground-truth annotations. Within the last four categories, examples are shown in pairs. Within each pair of examples, the upper example shows the suggested toxic spans based on the majority decisions made by the test context annotation (shaded in yellow) and the ground truth labeling decision by the annotators (bolded). In contrast, the lower example within each category shows where the consistency of annotation breaks compared to the upper example.}
    \label{tab:examples}
\end{table*}

\paragraph{Model Errors}

The first four categories (Category 1--4) in Table \ref{tab:examples} demonstrate the mistakes made by the \ptft model and ground-truth annotations. 
%

Category 1 shows where the \ptft model identifies valid toxic spans not present in ground truth, which accounts for 101 (50.5\%) of the model errors in the 200 sampled contexts. In these cases, annotators marked nothing as toxic in 58 contexts. However, most of the overlooked toxic spans are overly-common toxic words like Examples 491 and 1374 in Table \ref{tab:examples}. 

In comparison, in Category 2, there are 81 out of 200 sampled contexts with unmarked toxic span labels in the \ptft model output, in which the \ptft model produced an empty span or an incorrect span as toxic in four cases. Example 642 in Table \ref{tab:examples} shows a typical case were a relative rare toxic word, ``caca,'' is overlooked by the model. In the remaining cases, the \ptft model has shorter predictions than the ground truth toxic spans, matching the low performance on predicting longer toxic spans (\Sref{sec:pretraining}). 
%
In some cases, when multiple toxic spans exist in the same document, ground truth only labels one or two spans of them (e.g. Examples 1852 and 1486). While in other cases, ground truth would label more toxic spans (e.g. Example 346). The inconsistencies from ground truth annotations occur more when it comes to the multiple-word labeling, which will be described more in the next analysis for Category 5.

The missing labels from both scenarios are straight-forward in their formats and contexts. Besides, there are 20 contexts that both the ground-truth spans and \ptft model missed the toxic span partially or completely. 

Generally, ground truth annotation seldom labels non-toxic spans as toxic (Category 3). On the contrary, it is common for our \ptft models to make mistakes on labeling non-toxic spans (Category 4). It happens mostly in the cases when the \ptft model misinterprets the context (e.g. Examples 273 and 1802). 

\paragraph{Inconsistencies in Ground Truth Labels}

The last four categories (5--8) in  \tref{tab:examples} show common inconsistencies in annotation decisions, which we hope could aid in improving consistency in future work. 
%

In Category 5, the standard for spans labeling   is not consistent for which words are included in the toxic phrase. In some cases, when a sentence is fairly short ($<$ 5 words) and contains toxic words, the ground truth annotation would label out the adjective used for describing the trailing noun (e.g. Example 1469). However, in other cases (e.g. Example 773), the standard would change by skipping the adjectives. Moreover, this inconsistent annotation also occurs in the case when the underlying nouns are almost the same (e.g. Examples 1447 and 1776).

% These categories are also very correlated with model errors (Category 1 and 2. The problems become more obvious when it comes to the labeling for the common toxic words. 
Categories 6 and 7 comprise the majority of the inconsistencies in the annotation standards by the ground truth. These inconsistencies commonly manifest for frequent toxic words. For instance, in Examples 968 and 348, both of the "hypocrite(s)" should be toxic given the context and there is no more than one other toxic word within the document. The omission of common toxic words is the major source for this category. In many cases, the subtle variations of the document context would make it even harder to maintain a unified standard across different annotators. Hence, the introduction of some model-based labeling (or checking) could greatly improve the inconsistencies of this case.

In the sampled contexts, 25 documents consist of only one sentence. %Among them, Category 8 marks one of the most representative labeling patterns from the ground truth, 
Annotators varied in how much of these contexts to label (Category 8), occasionally marking the entire sentence as offensive (20\% of these single-sentence contexts), as in Example 894. %  17 of these 25 documents have less than 10 words in it, which makes them as short document. There are 5 cases the whole sentence is labeled as toxic like the Example 894. 
However, in a few cases (4 of 25), annotators labeled nothing as toxic (e.g. Example 1374). Interestingly, 9 of 25 cases where ground truth either labels the entire sentence or nothing, our \ptft model is able to identify the toxic word(s), suggesting the model is still effective for short contexts. %. In other words, \ptft model actually maintained a more stable standard when it comes to the short one-sentence context.



% Moreover, there are decent size of cases that ground-truth spans are inconsistent in the annotation practices. We provide some samples in the Table\ref{tab:examples_for_inconsistency}

\section{Discussion and Future Work}

Based on error analysis, our \ptft model suffers from low performance when generating predictions on non-toxic contexts or long toxic spans. A modified error function that rewards for edge-case scenarios can potentially improve the \ptft performance. Moreover, during the pre-training, we applied a simple-cutoff on the local weights to make labeling decisions for LIME explanations. The cutoff was determined solely based on evaluations with the Task data. If the LIME labeling could introduce more robust variants in the loss evaluation, the silver data span labeling might be more representative of the Task data's annotation logic.

Through comparing silver data with the gold data, we find the toxicity of some words is influenced by the broader linguistic environment. While the silver and gold data both consist of online comments, their time spans and topics are very different. The gold data uses contexts from 2015-2017 and has a concentration on political news; while the silver data covers 6 months of 2018 with no focused topics. Our qualitative analysis finds that the addition of non-toxic examples in the silver data influenced the model to consider overly common toxic words less toxic than they were in the gold data. Future work is needed to identify the optimal ratio of the toxic and non-toxic samples and to address domain/register differences in the data.
%during pre-training
%the first
%potentially

Last, the current approach could invite several natural improvements. For example, in the pre-training phase, we used TF-IDF embedding and logistic regression for the base of LIME explanations. This combination was chosen for its efficiency in the LIME training phase. However, many other embedding and model combinations rendered much better classification results, which may generate better rationales for pre-training. 
%Moreover, the current loss function for training our models could be extended to generate spans that match particular linguistic boundaries.

\section{Conclusion}

We presented our system for SemEval-2021 Task 5 on Toxic Span Prediction. Our initial approach used explainable machine learning (LIME) to generate a heuristically labeled span dataset, which was used to pre-train a RoBERTa model to recognize toxic spans. However, our results show that when fine-tuned on the task data, the resulting model generates slightly shorter explanations and ultimately performs slightly worse than a model trained only on the Task's training data---likely due to bias towards shorter spans generated by LIME. In our subsequent error analysis, we show that the majority of our model's errors (50.5\%) are associated with missed annotations in the ground truth, suggesting that actual model performance may be higher in practice.

\bibliography{references}
\bibliographystyle{acl_natbib}


\appendix

\section{Hyperparameter Settings}
\label{appendix:a}

Tables \ref{tab:Pre-trained Model parameters} and \ref{tab:Fine-tuning Model Parameters} respectively show the hyperparameter settings for the fine-tuned (\ft) model and the model pre-trained on silver data and then fine-tuned on the training data (\ptft). 

\begin{table}[tb]
  \centering
    \resizebox{0.49\textwidth}{!}{
    \begin{tabular}{r c}
        Hyperparameter & Value \\
        \hline
        learning rate                      & 5e-05 \\
        number of epochs                  & 1  \\
        weight decay              & 0    \\
        model      & RoBERTa-base  \\
        dropping probability of the dropout layer           & 0.5  \\
        learning rate scheduling & linear \\
        optimizer & AdamW \\
        warmup steps & 300\\
        random seed & 42 \\
        \end{tabular}
    }
  \caption{Hyperparameters of pre-trianed model}
  \label{tab:Pre-trained Model parameters}%
\end{table}%

\begin{table}[tb]
  \centering
    \resizebox{0.49\textwidth}{!}{
    \begin{tabular}{r c}
        Hyperparameter & Value \\
        \hline
        learning rate                      & 5e-5 \\
        number of epochs                  & 10  \\
        weight decay              & 1e-2    \\
        model      & RoBERTa-base  \\
        dropping probability of the dropout layer           & 0.5  \\
        learning rate scheduling & linear \\
        optimizer & AdamW \\
        warmup steps & 0\\
        random seed & 42 \\
        \end{tabular}
    }
  \caption{Hyperparameters of fine-tuned model}
  \label{tab:Fine-tuning Model Parameters}%
\end{table}%

\end{document}
