\section{Results and Discussion}
\label{sec:results}

\todo{Revise the discussion focusing on which conclusions can be drawn about the scorer (Sopan?)}

\begin{table}[t]
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{l|cccc|c}
\toprule
\textbf{Team} & \LIGHTS & \AMI   & \PERSUASIONS & \SWITCHBOARDS  & \textbf{Avg.} \\
\midrule
\multicolumn{6}{c}{\textbf{Eval AR}} \\
\midrule
Emory                   & \textbf{80.33} & \textbf{63.98} & \textbf{78.41}      & \textbf{74.49} & \textbf{74.3}\\
UTD\_NLP                & 79.56          & 57.38           & 77.50               & 72.64          & 71.8 \\
KU\_NLP                 & 69.16            & 57.59          & 71.09                & 65.67           & 65.9 \\
DFKI  & 64.99          & 43.93           & 59.93                & 53.55           & 55.6 \\
SCIR                    & 55.92           & 39.46           & 52.25                & 51.63           & 49.8 \\
\midrule
Baseline & 52.45           & 36.11           & 51.97               & 45.80           & 46.6  \\
\midrule
DFKI  & 61.26           & 00.00            & 59.20                & 51.24           & 42.9  \\
\bottomrule
\end{tabular}
}
\caption{{Performance on Task 1 (Evaluation Phase) -- Identity Anaphora (CoNLL Avg. F1)}}
\label{tab:task1}
\end{table}

\begin{table}[t]
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{l|cccc|c}
\toprule
\textbf{Team} & \LIGHTS & \AMI   & \PERSUASIONS & \SWITCHBOARDS  & \textbf{Avg.}  \\
\midrule
\multicolumn{6}{c}{\textbf{Eval Br (Gold)}} \\
\midrule
UTD\_NLP      & \textbf{19.73} & \textbf{19.65} & \textbf{31.40}      & \textbf{21.10} & \textbf{23.0} \\
KU\_NLP       & 16.67          & 15.30          & 18.79               & 18.33          & 17.3          \\
INRIA         & 9.35           & 6.00           & 16.28               & 7.79           & 9.9           \\
\midrule
Baseline & 6.35 & 6.21 & 13.77 & 5.39 & 7.9\\
\midrule
\multicolumn{6}{c}{\textbf{Eval Br (Pred)}} \\
\midrule
UTD\_NLP      & \textbf{13.98} & \textbf{13.33} & \textbf{21.92}      & \textbf{15.26} & \textbf{16.1} \\
KU\_NLP       & 13.46          & 10.25          & 12.32               & 10.99          & 11.8  \\
\midrule
Baseline & 6.01 & 4.94 & 9.34 & 3.78 & 6.0\\
\bottomrule
\end{tabular}
}
\caption{{Performance on Task 2 (Evaluation Phase) -- Bridging Anaphora (Entity F1)}}
\label{tab:task2}
% \vspace{-0.5cm}
\end{table}

\begin{table}[t]
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{l|cccc|c}
\toprule
\textbf{Team} & \LIGHTS & \AMI   & \PERSUASIONS & \SWITCHBOARDS   & \textbf{Avg.}  \\
\midrule
\multicolumn{6}{c}{\textbf{Eval DD (Gold)}} \\
\midrule
UTD\_NLP            & \textbf{43.44} & \textbf{36.91} & \textbf{52.09} & \textbf{40.44} & \textbf{43.2} \\
\midrule

\multicolumn{6}{c}{\textbf{Eval DD (Pred)}} \\
\midrule
UTD\_NLP            & \textbf{42.70} & \textbf{35.35} & \textbf{39.64} & \textbf{35.43} & \textbf{38.3} \\
DFKI & 20.97          & 17.43          & 23.76          & 23.86          & 21.5  \\
\midrule
Baseline & 12.12 & 15.75 & 18.27 & 13.55 & 14.9\\
\bottomrule
\end{tabular}
}
\caption{{Performance on Task 3 (Evaluation Phase) -- Discourse Deixis (CoNLL Avg. F1)}}
\label{tab:task3}
\vspace{-0.5cm}
\end{table}

In this section, 
we discuss the scorer's use in each of the tasks.

% we report the results of all systems submitted for each task.
%and discuss the differences among these approaches.

\subsection{Task 1 -- Identity Anaphora}
Task 1 saw the highest interest as five teams submitted a total of 36 runs
to the official leaderboard. The results on different sub-corpora are reported in Table~\ref{tab:task1}. 
Although only the CoNLL Avg. F1 scores were reported on the leaderboard, the scorer's setting utilized in this task allowed the organizers to provide additional details to participants about their systems' performances (Precision, Recall, and F1 scores) on multiple state-of-the-art metrics like {\BCUBED}, {\CEAF}, {\BLANC}, {\MUC}, and {\LEA}.

After the culmination of the eval-phase of the shared-task,

{
\begin{verbatim}
python ua-scorer.py key system \ 
    remove_singletons \ 
    remove_split_antecedent 
\end{verbatim}
}

\noindent a mode that is compatible with the Reference Coreference scorer was used by the organizers to evaluate systems only on coreferring markables. The analysis showed that every system lost about 5-8 CoNLL Avg. F1 points against their performance on the competition setting, across all four datasets. This reveals a slight bias the systems might have towards creating singleton clusters. 
Using the mode: %While, when

{
\begin{verbatim}
python ua-scorer.py key system \ 
    only_split_antecedent  
\end{verbatim}
}
\noindent %mode was deployed 
to isolate systems' performances on split-antecedents, the organizers found that even though the participating systems achieved high overall scores on Task 1, none of them were able to handle split-antecedents correctly, thus highlighting the need for further research in this direction.


% {\AMI} proved to be the toughest sub-corpus with the highest score being around 64\%, which is 10 percentage points below that of {\SWITCHBOARD}. 
% This is in line with the organizers' expectations, 
% as {\AMI} proved the most difficult corpus to annotate, with lots of uncertainty and ambiguity.
% Also, the conversations in {\AMI} are substantially longer than the other three datasets and hence require systems to consider long-distance relationships between mentions. Despite the differences in absolute performance on each sub-corpus, the best-performing system improved over the baseline by an impressive 30 CoNLL Avg. F1 percentage points.

\subsection{Task 2 -- Bridging Anaphora}
Three teams participated in Task 2 with \textit{INRIA} only participating in the gold mention setting. Entity F1 scores were reported for each sub-corpora.
% For the predicted setting Eval-Br (Pred), where the systems need to take a raw conversation as input during the inference time and perform both markable identification and bridging resolution, \textit{UTD\_NLP} performed best.
%the best across all four sub-corpora.
Precision/recall/F1 scores for other two metrics -- mention-based and anaphora recognition, were also output by the scorer to aid teams in evaluating different modules of their systems. Table~\ref{tab:task2} summarizes the performance of each team on this task. 

% \todo{Comments about using the scorer in  this task ...}
 
% Although both participating systems perform similarly across {\SWITCHBOARD}, {\AMI}, and {\LIGHT} (10 -- 15 Entity F1 points), \textit{UTD\_NLP} achieves an unusually high score on the {\PERSUASION} dataset (for this task), 
% reaching an Entity F1 of 21.92 percentage points. We see a similar trend in the gold setting (Table 2), where \textit{UTD\_NLP} and \textit{INRIA} score 8--10 absolute percentage points higher on {\PERSUASION} (31.40 and 16.28 respectively) as compared to their scores on the other three datasets.
% Finally, the performance of the runs in the gold setting is substantially higher than that in the predicted setting for both \textit{UTD\_NLP} and \textit{KU\_NLP} even though the gold setting does not distinguish between the markables that are relevant for the three tasks in the competition.

\subsection{Task 3 -- Discourse Deixis}

25 runs were received for Task 3. Two teams submitted to the predicted mention setting (Eval DD (Pred)) with \textit{UTD\_NLP} achieving performance around 35--42 CoNLL Avg. F1 percentage points on the different sub-corpora, almost doubling the score of the second team.
When %For the gold setting, where 
the gold markables were also released (Eval DD (Gold)), the system submitted by \textit{UTD\_NLP} managed a jump of more than 12 points on {\PERSUASION} (Table~\ref{tab:task3}).
% The performance of both teams on Task 3 is summarized in Table~\ref{tab:task3}. 

As discussed earlier, the scoring and metrics used for Task 3 were similar to that of Task 1. The command included an additional argument \CODE{evaluate\_discourse\_deixis} to only evaluate discourse deixis instances. The scorer also reported other state-of-the-art metrics like {\BCUBED}, {\CEAF}, {\BLANC}, {\MUC}, and {\LEA} for participants' reference.

Across all three tasks, the availability of different (scoring) modes and settings in the scorer allowed for a deeper understanding of the performance of different participating teams. 

% \subsection{Discussion}

% In retrospect, 
% organizing this shared task required tackling a few too many issues in a short time, from annotating the data to developing a new scorer to devising fair ways to use the scorer to assess systems, to be able to address all of them in a completely satisfactory way.
% This is why we decided to run the same task again next year, with new test data but maintaining the same genres and annotation guidelines. 
% % reusing this year's development and test sets as training/domain adaptation  and development sets, respectively. 
% The {\ARRAU} 3 data and annotation manual should also be available, addressing some of the concerns raised by the participants. 

% It would be  premature to infer too much about the tasks or the genres on the basis of this first experience, 
% but some interesting issues are already emerging from the analysis papers, such as the complex use of first and second person pronouns in these datasets. 
% We do hope it will be possible to carry out a more extensive analysis of the differences between these dialogue datasets and datasets based on written text and annotated for the same phenomena, such as the \CODE{RST} subcorpus of {\ARRAU}. 

% We would also like to congratulate all participants for gallantly tackling these relatively new tasks and new datasets yet  generally outperforming the baselines, some of  which were non-trivial. 