\section{Scoring Anaphoric Reference}
\label{sec:scoring}



\subsection{Scoring Identity Anaphora}

Evaluation is an issue with most areas of {\NLP}, but it has proven a  particularly difficult issue with identity anaphora, or coreference \cite{luo&pradhan:anaphora-book:evaluation}.
%\textcolor{gray}{A number of scoring methods have been proposed in the literature, from the link-based metric  known as {\MUC} scorer proposed by \CITEA{vilain-etal-1995-muc} to the mention-based {\BCUBED} metric \CITE{bagga-1998-bcubed} to the {\CEAF} metric \cite{luo-2005-ceaf}. }
%\todo{Add more details about these metrics if we have space}
%\textcolor{blue}{
There are various scoring methods for evaluating coreference resolution. {\MUC} \CITE{vilain-etal-1995-muc} is a link-based metric that computes a score based on the minimum number of additional or missing coreference links in the system output compared to the gold clusters. \BCUBED \CITE{bagga-1998-bcubed} is a mention-based metric that performs the evaluation based on the number of common mentions between system and gold coreference chains.
{\CEAF} \CITE{luo-2005-ceaf} is an entity-based metric that first finds the best alignment between the system and gold coreference chains and then computes the evaluation score based on the number of common mentions or the number of common links between the aligned chains.
{\BLANC} \CITE{recasens&hovy:blanc} is another linked-based metrics that considers both coreference and non-coreferring links for performing the evaluation.
Finally, {\LEA} \CITE{moosavi&strube:ACL16} is a linked-based entity-aware metric that performs the evaluations based on the number of common coreference links in each system and gold coreference chains. 
Since the  two {\CONLL} shared tasks \cite{pradhan2012conllst} it has become customary to score systems using the average F1 value of {\MUC},  {\BCUBED} and {\CEAF}, as originally proposed by \cite{denis&balridge:09}--this average, originally known as \ACRO{mela}, has since become known as the {\CONLL} metric.
%}
% \textcolor{gray}{
% As each of these metrics appears to capture some important intuition about identity anaphora,  since the  two {\CONLL} shared tasks \cite{pradhan2012conllst} it has become customary to score system using the average F1 value of {\MUC},  {\BCUBED} and {\CEAF}, as originally proposed by \cite{denis&balridge:09}--this average, originally known as \ACRO{mela}, has since become known as the {\CONLL} metric. 
% Much work  was invested in developing a reliable standardized scorer based on this metric (see next). 
% Further metrics were proposed by \CITEA{recasens&hovy:blanc} and by \CITEA{moosavi&strube:ACL16}.
% }

\subsection{The Reference Coreference Scorer}
During the period between the {\MUC} evaluations and the {\CONLLELEVEN} and {\CONLLTWELVE} shared tasks on coreference resolution, the reference implementation
% \footnote{In the form of an executable binary.} 
for only one evaluation metric---{\MUC}---was made available by the proposers of the metric itself.
Neither of the three intervening metrics came with  reference implementations.
To further complicate matters, only the article introducing the {\MUC} metric explicitly provided steps for computing the {\MUC} score for predicted mentions. This resulted in two fundamental misunderstandings across the research community in regards to the other three metrics:

\vspace{-5pt}
\begin{enumerate}
\item An assumption that both {\BCUBED} and {\CEAF} metrics could 
not handle %{\em not} {\em handle} 
predicted mentions and needed to be 
modified % {\em modified} 
in some way.
\vspace{-5pt}
\item 
Not realizing %{\em Not realizing} 
that the {\BLANC} metric was defined to handle 
only gold mentions. %{\em only gold mentions}.
\end{enumerate}
\vspace{-5pt}

The result were %What had resulted therefore 
multiple implementations of the metrics with  variations for scoring predicted mentions, and the  inaccurate computation of the {\BLANC} metric, for handling predicted mentions \cite{pradhan-etal-2014-scoring}.

The {\CONLLTWELVE} shared task used the first open source implementation of all scoring metrics created for the {\SEMEVALTWENTYTEN} shared task \cite{recasens-semeval-2010-coreference}. 
This was a significant step forward for the community in getting consistent evaluation scores across institutions (and therefore published articles). 
However, it was also built on top of the above two misunderstandings.

Soon after the conclusion of the {\CONLLTWELVE} shared task, these issues were uncovered. 
%For the sake of consistent coreference evaluations going forward, 
A committee of % a handful of 
coreference researchers---including almost all the original proposers of the existing metrics \MUC, \BCUBED, \CEAF, and {\BLANC}---
%worked together to resolve the misunderstandings and
created a open source, reference implementation
\footnote{\url{http://github.com/conll/ reference-coreference-scorers}}
%we reference to the link in section 4
for the research community \cite{pradhan-etal-2014-scoring},  which also included an extension of {\BLANC} to handle predicted mentions \cite{luo-etal-2014-blanc}. 
This became generally known as the Reference Coreference Scorer.

This implementation is however {\em only limited to evaluating coreference chains} and does not evaluate 
% the evaluation of 
other types of relations including non-identity anaphora, split-antecedent anaphora, bridging references, and discourse deixis. 

% \CITEA{pradhan-etal-2014-scoring} provide an open-source reference implementation for \MUC, \BCUBED, \CEAF, and {\BLANC} that has become known as the Reference Coreference scorer.
% This implementation addresses issues with the metrics and/or their implementation that resulted in inconsistent scoring when using predicted mentions. 
% This implementation is however only limited to evaluating coreference chains and does not evaluate the evaluation of other types of relations including non-identity anaphora, split-antecedent anaphora, bridging references, and discourse deixis. 

% %In addition, it does contain the implementation of the \LEA metric. 
% % \CITEA{pradhan-etal-2014-scoring} provide an open-source reference implementation for \MUC, \BCUBED, \CEAF, and \BLANC. This implementation addresses issues that resulted in inconsistent scoring when using predicted mentions. 
% % This implementation is however only limited to evaluating coreference chains and does not evaluate the evaluation of other types of relations including non-identity anaphora, split-antecedent anaphora, bridging references, and discourse deixis. In addition, it does contain the implementation of the \LEA metric. 
% \todo{Discuss here how the Reference Coreference scorer was developed starting from 2011 to 2014 (Sameer?)}

\subsection{Scoring Non-Identity Anaphora}

Unlike with identity anaphora, for which a generally accepted if not entirely satisfactory scoring mechanism has emerged,
%in the last ten years, 
no standards exist to evaluate system performance at the  other aspects of anaphoric interpretation, and most proposals are only concerned with evaluation on gold mentions.

\paragraph{Split-antecedent anaphora} There is limited 
%previous 
work on
split-antecedent anaphora resolution and its evaluation.
\CITE{vala-etal-2016-antecedents,yu-etal-2020-plural} only evaluate their models on split-antecedent anaphors, and on gold mentions only.
These methods 
compute %proposed a method for computing 
precision, recall, and F1 measures 
based on  the links between  split-antecedent gold anaphors and their  antecedent.
\CITEA{zhou&choi:COLING2018}  propose  %a method 
to evaluate split-antecedent resolution 
%the evaluation 
using the standard \ACRO{conll} scorer. 
They do this by adding the plural mention to each of the clusters for its atomic elements:
for example, they represent the \textit{\{\{John, Mary\}, They\}} entity as two  gold clusters--\{John, They\} and \{Mary, They\}.
This representation however violates the fundamental assumption behind the notion of coreference chain--that all mentions in a chain refer to the same entity.

\paragraph{Bridging references}
With bridging references we have a fairly clear idea on how to evaluate systems;  the main problem is that there is no complete agreement on the definition of bridging reference, and that many bridging references associate on more than one entity in context. 

Corpora annotated with bridging information may provide two types of information about a bridging reference: 
the entity the bridging reference is associated with, or \NEWTERM{anchor}, and  the most recent mention of this entity.
Many systems only carry out the second of these steps; some systems perform both. 
In early work \cite{vieira&poesio:CL} bridging descriptions were  evaluated by hand in order to take disagreements 
into account.
%disagreements between annotators.
\CITEA{poesio-et-al:ACL04} introduced \NEWTERM{entity evaluation}: a system's output is considered correct as long as the \textit{anchor} (i.e., the entity) is identified correctly, whether or not its most recent mention is.  
Unlike \CITEA{vieira&poesio:CL},  \CITEA{poesio-et-al:ACL04} required a system to identify the same anchor as in the gold 
even if the response 
%which was annotated even if the alternative 
was plausible. 
\CITEA{hou-et-al:CL18} introduced the more stringent \NEWTERM{mention evaluation}, which also requires a system to identify the exact mention of the anchor that is annotated in the corpus.

\paragraph{Discourse deixis} 
There has been  limited work on resolving discourse deixis and evaluating systems carrying out such task 
\cite{kolhatkar-et-al:CL18}.
In most annotations of discourse deixis  either a clause or a verb is marked as the antecedent of the discourse deixis, and most systems use some version of accuracy  to evaluate whether the system identified the correct antecedent. 
A more lenient metric,  \NEWTERM{Success@N}, was
proposed by Kolhatkar (e.g., \CITE{kolhatkar&hirst:EMNLP14}) and also
used by \CITEA{marasovic-etal-2017-mention} and in the {\CRAC} 2018 Shared Task \cite{poesio-etal-2018-anaphora}.
\ACRO{success@n}  is the proportion of instances
where the gold answer--the unit label--occurs within a system's first n choices.
(\ACRO{s@1} is standard precision.)

The approach to representing %event anaphora and 
discourse deixis adopted in Universal Anaphora however makes it possible however to adopt for discourse deixis the same approach to evaluation %typically
adopted in \NEWTERM{event coreference} \cite{lu&ng:IJCAI18}--namely, evaluate discourse deixis using the same metrics as entity coherence, thus assessing e.g., a system's ability to evaluate whole coreference chains started with a discourse deictic reference, but with a non-nominal first mention.
The approach to representing discourse deixis adopted in {\UA} makes it possible to adopt this  approach adopted in the {\UA} scorer;
as far as we know, this is the first time this approach has been used for discourse deixis.


\subsection{The CRAC 2018 Shared Task} 

The one previous shared task focused  on evaluating system performance at tasks other than identity anaphora was 
 the Shared Task on Anaphora Resolution with {\ARRAU} at {\CRAC} 2018 \cite{poesio-etal-2018-anaphora},
 which employed the {\ARRAU} corpus.
That shared task was articulated around three tasks: identity coreference (including identification of non-referring expressions), 
bridging references, 
and discourse deixis. 
The organization of the shared task resulted in the development of an extended version of the Reference Coreference Scorer %\cite{pradhan-etal-2014-scoring}, 
which also scores non-referring expressions. 
Separate scorers were developed for bridging reference resolution, carrying out both mention-based evaluation and entity-based evaluation of bridging references, as done by \CITEA{hou-et-al:CL18}, 
and for discourse deixis, based on \CITEA{kolhatkar&hirst:EMNLP14}.
The scorer presented in this paper integrates all these evaluations in a single scorer, adding the new ability to score split antecedent anaphora and an entirely new approach to discourse deixis evaluation.