\section{The CODI/CRAC 2021 Shared Task}
\label{sec:task}

The new {\UA} scorer was used as the official scorer for  
the %this year's 
{\CODICRAC} shared task. 
A brief description of the task is provided here 
to give %in order to provide 
some context for interpreting the scorer's results; for more details please see \cite{khosla-et-al:CODICRAC21}.

\subsection{The tasks}

Following the structure of the {\CRAC} 2018 Shared Task, 
{\CODICRAC} was articulated around three tasks covering 
%three key aspects of anaphoric interpretation: 
identity anaphora, bridging anaphora, and discourse deixis. Participants %or groups 
could submit to one or more tasks.  

% As customary, we ran two phases: \CODE{dev} (trial) and \CODE{eval}. 

\subsection{Gold and Predicted Settings}

% \todo{INCOMPLETE: Perhaps some more explanation of what these markables are - are they simply gold mentions? Also, the antecedents in discourse deixis are not among these markables, so your current description is not precise.}
Bridging reference resolution and discourse deixis are very difficult tasks. In consideration of this, the Bridging (Task 2) and Discourse Deixis (Task 3) tasks were further divided into system and gold settings, according to whether the markables would be predicted by the system or provided by the organizers. 
The two settings were run in order; the gold setting only became available after the runs under the system setting had been submitted. 
%This was done by the organizers taking into consideration the difficulty of the system markable setting. 
The two settings were scored separately.
%and independently.

\subsection{Settings of the UA Scorer used}

% \todo{Sopan, Juntao - check this ; This is correct! - Sopan}

%The {\UA} scorer can be run in a number of ways.
The following settings of the {\UA} scorer were used for the individual tasks.\footnote{For a full description of the task(s), see \url{https://github.com/sopankhosla/codi2021_scripts/blob/main/2021_CODI_CRAC_Introduction.md}}

\paragraph{Task 1}
%For Task 1, 
the evaluating coreference relations (including split-antecedents) and singletons modality was used. Non-referring expressions identification was not scored.

{\begin{verbatim}
python ua-scorer.py key system   
\end{verbatim}}

\paragraph{Task 2}  %For Task 2, 
the scorer was called using the command:
{
\begin{verbatim}
python ua-scorer.py key system \ 
    keep_bridging    
\end{verbatim}
}

\paragraph{Task 3}
%Finally, for Task 3, 
the scorer was called using the command:
{
\begin{verbatim}
python ua-scorer.py key system \ 
     evaluate_discourse_deixis    
\end{verbatim}    
}


% \subsection{Identity Anaphora}
% \todo{Massimo to take lead}

% \paragraph{Evaluation Script}

% To evaluate identity anaphora 
% The coreference evaluation
% script developed by Moosavi and Strube was modified to produce the scorer for Task 1. We will
% refer to this script as ’the extended coreference
% scorer’ below.3 The extended scorer, when run excluding non-referring expressions and singletons
% and ignoring MIN information, evaluates a system’s response using the same metrics (indeed, a
% reimplementation of the same code) as the standard CONLL evaluation script, v8 (Pradhan et al.,
% 2014).4 When required to use MIN information,
% the extended scorer follows the MUC convention,
% and considers a mention boundary correct if it
% contains the MIN and doesn’t go beyond the annotated maximum boundary. When singletons
% are to be considered, singletons are also included
% in the scores (all metrics apart from MUC can
% deal with singletons). Finally, when run in allmarkables mode, the script scores referring and
% non-referring expressions separately. Referring
% expressions are scored using the CONLL metrics;
% for non-referring expressions, the script evaluates
% P, R and F1 at non-referring expression identification. The extended coreference scorer is available
% from Moosavi’s github at https://github.
% com/ns-moosavi/coval

% \subsection{Bridging Anaphora}

% \paragraph{Evaluation Script}

% \subsection{Discourse Deixis}

% \paragraph{Evaluation Script} ...

% \subsection{Submission Details}
% \label{sec:submit}
% We used CodaLab to evaluate submissions and distribute the datasets. In the development phase, the participants only had access to out-of-domain training data (e.g., the {\ARRAU} corpus) and in-domain validation data. They could submit results to the public leaderboard to evaluate their systems. In addition, we also released the scoring script on Github to reduce the dependency on CodaLab during model development. During the evaluation phase, we released the unseen test set across four sub-corpora. The submissions were evaluated on each sub-corpus individually and the final ranking for each task was performed by taking the mean of the four scores. Due to the lack of in-domain training data, the participants were allowed to use additional resources.

% \subsection{Baselines}
% \label{sec:baseline}

% \todo{This Section could be cut to save some space}

% One baseline system was released for each task. 
% The baselines for identity and bridging anaphora were derived from current state-of-the-art methods;
% a simple yet effective
% heuristic method 
% was used
% for discourse deixis.

% For Task 1, \citet{xu2020revealing}'s coreference resolution model was used, but without any higher-order inference.\footnote{\url{https://github.com/lxucs/coref-hoi/}} We used the {\ONTONOTES} (English) dataset for training and development. This model is then evaluated on {\CODICRAC} datasets. %for final scores. 

% For Task 2, the baseline was derived from \citep{yu-etal-2020-bridging}. 
% The single-task variant was used, that is only trained on bridging annotations.
% % \todo{Did we use the single-task version of this architecture as the baseline? Yes. - Resolved Above}
% Their best-performing model,  trained on the 
% \ACRO{rst} sub-corpus of {\ARRAU}, was tested on {\CODICRAC} data.\footnote{\url{https://github.com/juntaoy/dali-bridging}} 

% The baseline for Task 3 leverages a simple heuristic that only considers demonstrative pronouns (\textit{this}, \textit{that}) as anaphors and considers the immediately preceding clause/utterance in the conversation to be their antecedent. Although simple, the algorithm achieves respectable scores on the {\CODICRAC} development corpus. The performance of each baseline on different sub-corpora is shown in Tables~\ref{tab:task1},~\ref{tab:task2}, and ~\ref{tab:task3}.

% To help participants interested in building upon these baselines, we released the scripts used in the baseline pipeline to convert the {\CODICRAC} data from
% the {\CONLL}-{\UA} format 
% %Universal Anaphora 
% to JSON structure and vice-versa for each task.\footnote{The necessary scripts are available from \url{https://github.com/sopankhosla/codi2021_scripts}} 
% The scripts contain modules that can also be used independently to transform conversations into a format compatible with different transformer-based encoders, allowing participants to set optional arguments to specify the \textit{segment\_size} and \textit{tokenizer} for the conversion. 

% \todo{Talk about the script to convert from UA to json}

% \section{Participating Systems}

% However, only \textit{UTD\_NLP} participated in all three tasks. We received system description papers from all 5 teams. 