\documentclass[11pt,a4paper]{article}
\usepackage{authblk}
\usepackage[hyperref]{acl2021}
\usepackage{times}
\usepackage{latexsym}
\usepackage{booktabs}
\usepackage{enumitem}
\usepackage[hyphens]{url}
\usepackage{multirow}
\renewcommand{\UrlFont}{\ttfamily\small}


\makeatletter
\g@addto@macro{\UrlBreaks}{\UrlOrds}
\makeatother

\usepackage{graphicx}
\graphicspath{ {./images/} }

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{***} %  Enter the acl Paper ID here

%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.

\newcommand\BibTeX{B\textsc{ib}\TeX}

\title{SkoltechNLP at SemEval-2021 Task 5: Leveraging Sentence-level Pre-training for Toxic Span Detection}

\author{\textbf{David Dale}$^\ddag$, \textbf{Igor Markov}$^{\ddag,\dag}$, \textbf{Varvara Logacheva}$^\ddag$, \textbf{Olga Kozlova}$^\dag$, \\\textbf{Nikita Semenov}$^\dag$, \textbf{and} \textbf{Alexander Panchenko}$^\ddag$}


\affil{$^\ddag$Skolkovo Institute of Science and Technology, Moscow, Russia\\
$^\dag$Mobile TeleSystems (MTS), Moscow, Russia}

\affil{\href{mailto:a.panchenko@skoltech.ru}{\{d.dale,igor.markov,v.logacheva,a.panchenko\}@skoltech.ru} \\\href{mailto:a.panchenko@skoltech.ru}{\{oskozlo9,nikita.semenov\}@mts.ru}}




\date{}

\begin{document}
\maketitle
\begin{abstract}
This work describes the participation of the Skoltech NLP group team (Sk) in the Toxic Spans Detection task at SemEval-2021. The goal of the task is to identify the most toxic fragments of a given sentence, which is a binary sequence tagging problem. We show that fine-tuning a RoBERTa model for this problem is a strong baseline. This baseline can be further improved by pre-training the RoBERTa model on a large dataset labeled for toxicity at the sentence level. While our solution scored among the top 20\% participating models, it is only 2 points below the best result. This suggests the viability of our approach. 
\end{abstract}


\section{Introduction}

Toxicity and offensive content is a major concern for many platforms on the Internet. Therefore, the task of toxicity detection has attracted much attention in the NLP community~\cite{wulczyn2017ex, hosseini2017deceiving, dixon2018measuring}. Until recently, the majority of research on toxicity focused on classifying entire user messages as toxic or safe. However, the surge of work on text de-toxification, i.e., editing of text to keep its content and remove toxicity~\cite{santos2018fighting, tran2020towards}, suggests that localizing toxicity within a sentence is also useful. If we know which words of a sentence are toxic, it is easier to ``fix'' this sentence by removing or replacing them with non-toxic synonyms. \newcite{DBLP:journals/corr/abs-2012-10289} make human labelers annotate the spans as rationales for classifying a comment as hateful, offensive, or normal. They show that using such spans when training a toxicity classifier improves its accuracy and explainability and reduces unintended bias towards toxicity targets.

This year the SemEval hosts the first competition on toxic spans detection, namely, SemEval-2021 Task 5\footnote{\url{https://competitions.codalab.org/competitions/25623}}~\cite{pav2020semeval}. It provides training, development, and test data for English. 
As far as we know, it is the first attempt to explicitly formulate toxicity detection as sequence labeling instead of classification of sentences.


Multiple NLP tasks recently benefited from transfer learning --- transfer of probability distributions learned on some task to another model solving a different task. The most common example of transfer learning is the use of embeddings and language models pre-trained on unlabeled data (e.g. ELMo~\cite{peters-etal-2018-deep}, BERT~\cite{devlin-etal-2019-bert} and its variations, T5~\cite{raffel2020exploring}, etc.) on other tasks (e.g. \newcite{he2021deberta,wang2019structbert} \textit{inter alia} use pre-trained BERT models to perform tasks from the GLUE benchmark~\cite{wang-etal-2018-glue}). 

Word-level toxicity classification can be formulated as a sequence labeling task, which also actively uses the pre-trained models mentioned above. BERT comprises the versatile information on words and their context, which allows to successfully use it for sequence labeling tasks of different levels: part-of-speech tagging and syntactic parsing~\cite{koto-etal-2020-indolem}, named entity recognition~\cite{hakala-pyysalo-2019-biomedical}, semantic role labeling~\cite{he-etal-2019-syntax}, detection of Machine Translation errors~\cite{moura-EtAl:2020:WMT}. 

This diversity of applications suggests that word-level toxicity detection can also benefit from pre-trained models. Besides that, toxicity itself has been successfully tackled with BERT-based models. Research on sentence-level toxicity extensively used BERT and other pre-trained models. Both language-specific and multilingual BERT models were used to fine-tune toxicity classifiers~\cite{leite-etal-2020-toxic,ozler-etal-2020-fine}. This shows that BERT has information on toxicity.

%We use the same technique. 
Thus, we follow this line of work. 
Namely, we fine-tune a RoBERTa model~\cite{roberta} to perform a sequence labeling task. Besides that, we train a model for sentence classification on the Jigsaw dataset of toxic comments and use the information from this model to detect toxicity at the subsentential level. This helps us overcome the insufficient data size.  

This is in line with previous work, which has shown that sentence-level labels can be used in combination with token labels~\cite{rei2019jointly} or completely substitute them~\cite{rei2018zero, schmaltz2019detecting}. 

In our experiments, we test the hypothesis that the sentence-level toxicity labeling can be used for a sequence labeler that recognizes toxic spans in text. We suggest three ways of incorporating this data: as a corpus for pre-training, pseudo-labeling, and for joint training of sentence-level and token-level toxicity detection models. Our experiments show that the latter method yields the best result. Moreover, we show that using sentence-level labels can dramatically improve toxic span prediction when the dataset with token-level labels is small. 

The contributions of this work are the following:
\begin{itemize}%[noitemsep]
    \item We successfully use the dataset labeled for toxicity at the sentence level for token-level toxicity labeling,
    \item We propose a model for joint sentence- and token-level toxicity detection,
    \item We analyze the performance of our models, showing their limitations and reveal the ambiguities in the data.
\end{itemize}

%\section{Related work}

%Some related work on sequence tagging and toxic text analysis. I guess the intersection is void; hence this task was created, but it makes sense in my opinion to mention a few references from both of them briefly. 

% A few papers, such as \cite{DBLP:conf/www/NobataTTMC16, DBLP:conf/acl-alw/ParkF17, DBLP:conf/icwsm/DavidsonWMW17, wulczyn2017ex, DBLP:conf/naacl/ZampieriMNRFK19} work on classification of toxic comments in social networks. Some of them further divide toxic comments into subclasses by type of toxicity, such as hate speech or offensive language, or by target of toxicity, such as sexism or racism. \cite{dixon2018measuring} investigate the performance of toxicity classifiers with regard to various identity groups as targets of toxicity. However, all these works treat a comment as an indivisible unit that can be entirely toxic or not. \cite{DBLP:journals/corr/abs-2012-10289} go further by annotating the spans which, according to annotators, are important for classifying the comment as hateful, offensive, or normal and demonstrate that using these spans to guide model attention during training improves classification performance and reduces unintended bias. However, they do not try to explicitly determine which spans are actually toxic. 

\section{The task}

The training data of the task comprises 7,940 English comments with character-level annotations of toxic spans. The labeling was performed manually by crowd workers.

The spans labeled as toxic often contain rude words: ``\textit{Because he's a \underline{moron} and a \underline{\smash{bigot}}. It's not any more complicated than that.}'' (toxic spans are \underline{underlined}). Other toxic spans consist of words that become toxic in context: ``\textit{Section 160 should also be amended to include \underline{sexual acts with animals} not involving penetration}''. Borders of some toxic spans fall in the middle of a word; we treat such cases as markup errors.

As a development set, we use the trial dataset of 690 texts provided by the task organizers. We evaluate our final models on the hidden test set of the task consisting of 2,000 texts.

\section{Pre-training for toxic span detection}

Here we give the motivation behind our models and describe their architecture and training setup.

\subsection{Motivation}

Our intuition is that the toxicity is often lexically-based, i.e., there are certain words that are considered offensive and make the whole sentence toxic. In this case, we expect that as we add extra data to our toxic span dataset, after some point, the vocabulary of toxic words in it will saturate and stop increasing. However, Figure \ref{image:vocab} shows that the size of the toxic vocabulary linearly depends on the dataset size, which suggests that its size is insufficient for the task. In this case, the model will often need to label unseen words. To mitigate the lack of data, we leverage the additional dataset with toxicity information, namely, the Jigsaw toxic comments dataset\footnote{\url{https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data}} which features 140,000 user utterances labeled as toxic or safe.

\begin{figure}[ht]
\includegraphics[width=6.5cm]{vocab}
\centering
\caption{\label{image:vocab} Size of the toxic vocabulary as a function of the corpus size.}
\end{figure}

\subsection{Transfer learning for spans detection}
\label{sec:models}

Our base model is RoBERTa. We fine-tune \texttt{roberta-base} model on the toxic spans training set for sequence labeling task (further denoted as \textbf{RoBERTa tagger}). This model already gives promising results. 
We further improve it by providing it with the additional training signal from the Jigsaw toxic comments dataset.
We propose three ways of incorporating this data.

The first way is \textbf{pseudo-labeling} \cite{pseudo}. 
We apply the \textbf{RoBERTa tagger} to predict the toxic spans in the Jigsaw dataset. We use these predictions to further train the model. 

Another option is to use the Jigsaw data to fine-tune RoBERTa with it. We suggest two scenarios. The first is to fine-tune the model on the Jigsaw dataset for the sentence classification task, and then on the toxic spans dataset --- this model is referred to as \textbf{RoBERTa classifier + tagger}. In this case, the model has different output layers for the two tasks, and other layers are shared.

Finally, we propose a novel architecture for joint token- and sentence-level classification, where the score \textbf{y} for a sentence $\textbf{x} = \{x_1, x_2, ..., x_n\}$ is computed as the average of word-level scores:

$$\hat{\textbf{y}}=\sigma \left( \alpha + \beta \frac{1}{n}\sum_{i=1}^n \hat{y}_i^\gamma \right)$$

where $\alpha$, $\beta$ and $\gamma$ are trainable parameters, and $\sigma$ is the logistic function. This model does not need to be trained on the data with token-level labeling but can get token-level toxicity information from sentence labels. 
We fine-tune this model both on Jigsaw and toxic spans datasets. The model is referred to as \textbf{tagging classifier}.

\subsection{Working with spans}
To reformulate toxic spans detection as a token classification problem, we label a token as toxic if at least one of its characters is toxic. When projecting the predicted token-level labels back to the character level, we try two strategies:

\begin{enumerate}[noitemsep]
    \item Consider a token to be toxic if its toxicity score is higher than the threshold, do not force the labels of tokens within a word to agree with each other.
    \item Consider a word to be toxic if the aggregated toxicity score of all its tokens is higher than the threshold. We try four different aggregation functions: min, max, mean, and the simplified naive Bayes formula: $$\tilde{x}=\frac{\prod_i{x_i}}{\prod_i{x_i} + \prod_i{(1-x_i)}}.$$ 
\end{enumerate}
In both methods, we label a space character as toxic only if the characters both to the right and to the left of it are toxic.

\section{Baselines}

In this section, we present a set of common baseline approaches used for sequence tagging, such as CRF and LSTM with pretrained word embeddings. We implement them in order to analyze the performance of our methods in the context of other techniques.

\paragraph{Word-based LogReg}

This is a vocabulary-based method: we label words as toxic if they appear in our toxic vocabulary. The vocabulary is created as follows. We create a set of toxic and safe phrases, where toxic phrases are toxic spans from our data and safe phrases are sentences from our data with removed toxic spans. We then train a binary logistic regression classifier of toxic and safe phrases using words as features. The by-product of this classifier is the list of weights for all words from the data. We consider words with weights greater than a threshold as toxic.


\paragraph{Attention-based LogReg}

Another approach to represent words is to take their attention weights from a RoBERTa-based sentence-level toxicity classifier (we train it on the Jigsaw dataset). We assemble attention weights from all RoBERTa heads and layers in a single vector of dimension 144. These vectors are used as features in a logistic regression classifier. This approach is motivated by the fact that a RoBERTa model trained to recognize toxicity puts more emphasis on certain words associated with sentence-level toxicity. Surprisingly, this model underperforms the logistic regression classifier, which uses words as features.


\paragraph{Conditional Random Fields}

We suggest that the toxicity level of a word can be context-dependent, so we also experiment with sequence labeling models. We try Conditional Random Fields (CRF) \cite{lafferty2001conditional} model. It uses the following features: the word itself, the word's part of speech, whether the word is a digit and consists of uppercase letters. Each word is represented with these features of the current, previous, and next words. The model performs closely to the attention-based classifier.


\paragraph{Sequence labeling with LSTM}

Finally, we experiment with the LSTM architecture \cite{hochreiter1997long}. We implement a Bi-LSTM network and also train an LSTM tagger from the AllenNLP library.\footnote{\url{https://github.com/allenai/allennlp}} We do not use any pre-trained embeddings in the Bi-LSTM model and use two versions of the AllenNLP LSTM: without pre-trained embeddings and with GloVe embeddings \cite{pennington2014glove}. 



\section{Evaluation}
%In this section, we present the experimental setup and our results.


\subsection{Experimental Setting}
\label{section:setting}

For each transfer learning model, we use two-stage fine-tuning. We first train only the output layers of the models with the learning rate of $10^{-3}$, and then the whole models with the learning rate of $10^{-5}$. In both cases, we use linear learning rate warm-up for 3000 steps. We use the AdamW optimizer~\cite{LoshchilovH19} and the batch size of 8, and early stopping to determine the number of training steps. We use the \texttt{transformers}\footnote{\url{https://huggingface.co/transformers}} library for training. 

For all the models which use the additional data from Jigsaw, we apply this scheme twice. The \textbf{pseudo-label} model is first fine-tuned on the original toxic spans dataset, and then on the self-labeled Jigsaw dataset, whereas the \textbf{RoBERTa classifier + tagger} and \textbf{tagging classifier} are first fine-tuned on the Jigsaw dataset (as classifiers), and then on the toxic spans dataset (as taggers).


\subsection{Results}

The scores of our models and the competing systems are shown in Table \ref{table:finetune}.
Our best submitted system (\textbf{tagging classifier}) had the F$_1$-score of 0.681 on the test set, while the best team over the whole task got 0.708. This brings our team to the top 20\% of the leaderboard. The pseudo-labeling approach was only marginally worse, scoring 0.674. Simply fine-tuning RoBERTa only on the tagging problem scored 0.668. On the other hand, none of our baselines could approach this result. Our best baseline is the word-based LogReg classifier. Apparently, other models fail to learn even the toxic vocabulary because their word representations are not informative enough.


\begin{table}[ht]
\centering
\begin{tabular}{lr}
\toprule \multicolumn{1}{c}{\bf Model} & \multicolumn{1}{r}{\textbf{F$_1$ score}} \\ \midrule
 %\textbf{} & \textbf{at 0.5} & \textbf{at 0.6} \\ \midrule
\multicolumn{2}{c}{\bf Top-5 participants} \\
\midrule
HITSZ-HLT	& 0.708 \\
S-NLP	& 0.707 \\
hitmi\&t	& 0.698\\
L	& 0.698  \\
YNU-HPCC	& 0.696\\ \midrule
% Official baseline & 0.690 \\ \midrule % Pavlopoulos  says this is just the name of a participant
\multicolumn{2}{c}{\bf Our models} \\ \midrule
tagging classifier &  0.683 \\   % 
pseudo-labeling      & 0.682 \\
RoBERTa tagger     & 0.678 \\
RoBERTa classifier + tagger  & 0.670 \\ \midrule
\multicolumn{2}{c}{\bf Our baselines} \\ \midrule
Word-based LogReg                & 0.556 \\
 LSTM basic embeddings           & 0.538 \\
 Bi-LSTM basic embeddings        & 0.530 \\
 Attention-based Logreg   & 0.524     \\
 CRF                            & 0.523 \\
 LSTM Glove embeddings           & 0.497\\
\bottomrule
\end{tabular}
\caption{\label{table:finetune} Performance of our models (baselines and RoBERTa-based models) and their comparison with the 5 best-performing participants. Models within each section are sorted from best to worst.}
\end{table}

While our best-performing model is only 18th best out of 92 participating systems, the results of the top systems are fairly close to ours (the difference is less than 2.5\%). The variation of deep learning models often falls in this margin \cite{reimers2017reporting}. For our models, the sample standard deviation of the F$_1$-score is about 0.9\%, so the difference between their performance is likely to be statistically insignificant. 

\begin{table}[ht]
\centering
\begin{tabular}{lr}
\toprule \textbf{Aggregation Type} & \textbf{F$_1$ score} \\ \midrule
No aggregation  & 0.685  \\ \midrule
\multicolumn{2}{c}{\bf Aggregation of word-level scores} \\ \midrule

max token score            & 0.673  \\
min token score            & 0.641  \\
average token scores           & 0.670  \\ 
naive Bayes     & 0.653  \\
\bottomrule
\end{tabular}
\caption{\label{table:decoding} Scores of the \textbf{tagging classifier} model with different token aggregation methods (computed on the development set).} 
\end{table}

An important hyper-parameter of the models is the probability threshold. It is usually fine-tuned on the development set. However, the development set provided for the task is too small. The threshold fine-tuned on it performs even worse than the standard threshold of 0.5. Thus, during the evaluation period of the competition, we tried submitting models with different threshold values. While this is not a completely fair practice because we indirectly used the test for tuning a model parameter, we suspect that many teams were overfitting to the test set in a similar way. We suggest that in order to make the evaluation fair, the results of the models on the final test set should not be available before the end of the competition, even in the indirect way (i.e., in the form of teams ranking without scores as it was done in this competition).

\begin{table}[]
    \centering
    \begin{tabular}{l|rr}
    \toprule
\multicolumn{1}{c|}{\multirow{2}{*}{\bf Model}} & \multicolumn{2}{c}{\bf threshold} \\
\multicolumn{1}{c|}{}                       & \multicolumn{1}{c}{\bf 0.5} & \multicolumn{1}{c}{\bf 0.6} \\ \midrule
tagging classifier & 0.681 & \textbf{0.683} \\   % 
pseudo-labeling       & 0.674 & 0.682 \\
RoBERTa tagger     & 0.668 & 0.678 \\
RoBERTa classifier + tagger  & 0.664 & 0.670 \\ \bottomrule
    \end{tabular}
    \caption{F$_1$-scores of models with different probability thresholds.}
    \label{tab:result_thresholds}
\end{table}

%\footnote{We suspect that we were not the only team overfitting the leaderboard in this task. We advise task organizers to split the test set into public and private parts in order to motivate participants to tune parameters with cross validation instead.} 
The best results which we report here were achieved with the threshold of 0.6 (see Table~\ref{table:finetune}). We compare these results with those of the same models with the default threshold of 0.5 in Table~\ref{tab:result_thresholds}. It shows that these scores are lower by up to 1\%.


Another hyper-parameter of our models is the method of converting token-level labels to character-level labels. We compare different methods on the development set (see Table \ref{table:decoding}). 
Surprisingly, the prediction of labels for each token individually with no aggregation works better than assigning labels to the whole words. This might happen because the attempts to decode words consistently lead to the propagation of wrongly predicted labels. Following this observation, we use the no-aggregation strategy for all models.




\begin{table*}[htb]
\centering
\begin{tabular}{p{1.0\linewidth}}
\toprule
\textbf{Correct labeling} \\ \midrule
See a shrink you \underline{\smash{\textbf{pathetic troll}}}. \\ 
They're not patriots. They're \underline{\textbf{vandals}}, \underline{\textbf{thieves}}, and \underline{\textbf{bullies}}. \\ 
Trudeau and Morneau are fiscally and economically \underline{\smash{\textbf{inept}}} and \underline{\smash{\textbf{incompetent}}}. \\ 
\midrule
\textbf{Incorrect labeling} \\ \midrule
That's right. They are not normal. And I am starting from the premise that they are \underline{ABNORMAL}. Proceed wth the typical racist, \textbf{bigot, \underline{sexist rubbish}}. Thanks! \\ 
ADN is endorsing, without officially endorsing. \underline{Bunch of \textbf{cowards}}!!! \\
Rabidly \textbf{anti-Canadian} \underline{troll}. \\
\hline
\end{tabular}
\caption{\label{table:text-examples} Examples of ground truth (\underline{underlined}) and predicted (\textbf{in bold}) toxic spans}
\end{table*}



\subsection{Efficiency of pre-training}

To understand the effect of the use of additional sentence-labeled data, we compare the performance of \textbf{RoBERTa tagger} (a model which uses only the toxic span dataset) and \textbf{tagging classifier} (a model which uses sentence-labeled Jigsaw data in addition to the toxic span dataset) models trained on subsets of the data of different sizes. We would like to see if the usefulness of additional sentence-labeled data reduces as we get more data with token-level labeling.


Figure \ref{image:curve} plots the F$_1$-scores of the two models trained on datasets of sizes between 10 and 7,940 sentences. It shows that when the training set size is between 10 and 1,000, pre-training with sentence-level annotations gives a considerable boost in performance. However, the effect of this pre-training becomes insignificant after the size of the data with word-level labeling reaches around 3,000. Thus, this pre-training strategy is efficient only in cases when the size of the data with word-level labeling is very small.

\begin{figure}[ht]
\includegraphics[width=6.5cm]{curve}
\centering
\caption{\label{image:curve} Learning curves for two transfer learning models, with (\textbf{tagging classifier}) and without (\textbf{RoBERTa tagger}) additional sentence-level data.}
\end{figure}

\begin{table*}[ht]
\centering
\begin{tabular}{p{0.2\linewidth}p{0.2\linewidth}|p{0.2\linewidth}p{0.2\linewidth}}
\toprule
 \multicolumn{2}{c|}{\textbf{Top 20 false positive words}} & \multicolumn{2}{c}{\textbf{Top 20 false negative words}} \\
\midrule
stupid & ass & and & of \\
idiot & liar & the & have \\
ignorant & garbage & are & loser \\
moron & loser & a & crap \\
dumb & fools & ignorant & all \\
idiots & troll & racist & chemical \\
fool & crap & you & that \\
pathetic & damn & in & not \\
stupidity & fuck & is & bunch \\
ridiculous & clown & to & dumb \\
\bottomrule
\end{tabular}
\caption{\label{table:word-examples} The most common false positive and false negative words}
\end{table*}


\subsection{Error analysis}
We analyze the errors of our best submitted system (\textbf{tagging classifier}) by comparing its predictions with the ground truth labels released after the end of the competition.

The vocabulary of false negative spans (527 unique tokens) is more diverse than that of false positives (275 unique tokens), while the number of false positives and false negatives in the test set is comparable (860 vs 813 tokens). It may indicate that the model is cautious and prefers to highlight only the hypotheses which have high confidence, % the sure options, 
while human annotators are more creative in their analysis. We give some examples of correct and incorrect labelings by our model in Table~\ref{table:text-examples}.



The most frequent false positive words characterize incompetence or lack of mental capacities: \textit{stupid}, \textit{idiot}, \textit{ignorant}, \textit{moron}, \textit{dumb}, etc. Other frequent false positives are derogatory (\textit{pathetic}, \textit{ridiculous}, \textit{ass}, \textit{garbage}, \textit{loser}, etc.), denounce particular misdeeds (\textit{liar}, \textit{troll}, \textit{racist}, \textit{hypocrite}, etc.), or express general negativity (\textit{damn}, \textit{fuck}, etc.). It is not obvious why human annotators label them as toxic in some cases, and as non-toxic in other cases. We suspect that inter-annotator agreement on such words is not very high.

The most frequent false negative words are function words: \textit{and}, \textit{the}, \textit{are}, \textit{a}, \textit{you} etc. It happens because annotators sometimes label the whole text or a large chunk of it as toxic. The more meaningful false negatives belong to the same classes as the false positives (\textit{ignorant}, \textit{racist}, \textit{loser}, etc.). The most common false positive and false negative words are listed in Table~\ref{table:word-examples}.



In general, the performance on this task might be limited to 0.7 F$_1$-score (the quality of the best-performing model) % in the competition) 
by the ambiguity of the annotations. In future work, it %may be worthwhile to refine is worth refining the annotations using a more rigorous definition of toxicity or to distinguish between the degrees of toxicity instead of viewing it in a binary way.

\section{Conclusions}


We present a number of models for the detection of toxic spans within toxic sentences. All models are RoBERTa language models fine-tuned on the data with character-level labeling of toxic spans. In addition to that, we perform fine-tuning on an additional dataset with sentence-level toxicity labeling. This yields an improvement. However, our analysis shows that the effect of such pre-training is marginal when the main dataset size exceeds 1,000 samples. Therefore, substantial improvement is observed for small dataset sizes. Nevertheless, the models we propose can be useful in extremely low-resource scenarios.

Our model performs closely to the winning systems. We suggest that the differences between the 20 top models might be attributed to the variation of deep learning models and overfitting the test set.
In addition to that, the error analysis shows that some errors in our model might be due to inconsistencies in the test data.

We release the code required to reproduce our experiments online.\footnote{\url{https://github.com/skoltech-nlp/toxic-span-detection}} 


\section*{Acknowledgements}

This work was conducted in the framework of the joint Skoltech-MTS laboratory. 

\bibliographystyle{acl_natbib}
\bibliography{anthology,acl2021}

%\newpage
%\appendix
%\onecolumn


%\section{Appendix: Most frequent errors}
%\label{app:wrong_words}



\end{document}
