\section{The Universal Anaphora Scorer}
\label{sec:scorer}

% \todo{JY: I did the first pass on this, let me know if we need more details. MP: Thanks Juntao - as this is the core section of the paper it would be good to beef it up a bit - e.g., providing some examples of the scorer on various aspects of anaphoric interpretation?}

The new Universal Anaphora (\UA) scorer is a Python scorer for the varieties of anaphoric reference covered by the Universal Anaphora guidelines, which include identity reference, split antecedent plurals, identification of non-referring expressions, bridging reference, and discourse deixis.

The scorer builds on the original Reference Coreference scorer
\footnote{\url{https://github.com/conll/reference-coreference-scorers}} 
\CITE{pradhan-etal-2014-scoring} developed for use in the {\CONLL} 2011 and 2012 shared tasks on the {\ONTONOTES} corpus \CITE{pradhan2012conllst}
and its reimplementation in Python by Moosavi,\footnote{\url{https://github.com/ns-moosavi/coval}} developed for
%which was already extended to evaluate non-referring expressions evaluation and cover singletons for 
the {\CRAC} 2018 shared task \CITE{poesio-etal-2018-anaphora}. 

%% MOVED EARLIER
%% There are two Universal Anaphora (\UA) formats that have been proposed: the `compact' and the `exploded' formats. The `compact' format encodes all the anaphoric interpretations in the `\ACRO{misc}' column of the {\CONLL}-U format, this makes it fully compatible with the Universal Dependencies project. As a result, the resources collected for the {\UA} can also be used by the Universal Dependencies community. The `exploded' format is also based on the {\CONLL}-U format, instead of putting everything in the {\ACRO{misc}} column, it adds additional columns to accommodate different types of anaphoric information. The format has a focus on the {\UA} community, to make it easy to intemperate by humans and systems for anaphora resolution. The Universal Anaphora scorer currently supports the `exploded' format and is planning to be extended to support the `compact' format in the future.



\subsection{Identity Reference}
In `exploded' format,  identity reference (cluster id) is specified in the {\CODE{Identity}} column, which includes both singular clusters (including singletons) and split-antecedents. 
Parentheses are used to specify the boundaries of the mention,
as in the {\CONLL} format, and a set of attributes is attached to the opening parentheses to specify the annotations. 
This includes the cluster id (\CODE{EntityID}), markable id (\CODE{MarkableID}), 
the minimum span %(head) 
(\CODE{Min}) and the 
semantic type (\CODE{SemType}) (non-referring, new, old)
%information status (SemType) 
of the mention. 
Split-antecedent information is annotated on the antecedents's row using an `\CODE{ElementOf}' attribute that specifies the cluster id of the split antecedent plural anaphor. 
The following is an example of the {\ACRO{identity}} column:

{
\begin{verbatim}
(EntityID=10|\
MarkableID=markable_11|\  
Min=5|\ 
SemType=do|\ 
ElementOf=23)
\end{verbatim}
}

The scorer computes all major metrics for identity reference including 
{\MUC} \CITE{vilain-etal-1995-muc}, 
{\BCUBED} \CITE{bagga-1998-bcubed},
{\CEAF} \CITE{luo-2005-ceaf}, 
{\CONLL} (the unweighted average of {\MUC}, {\BCUBED}, and {\CEAF}) \CITE{pradhan-etal-2014-scoring}, 
{\BLANC} \CITE{luo-etal-2014-blanc,recasens&hovy:blanc}, 
and {\LEA} \CITE{moosavi&strube:ACL16} scores. 
\todo{Juntao, I wasn't sure what the previous text meant, is this it?}
The scorer preserves the settings used in the Reference Coreference scorer for the {\CONLL} shared tasks,
and its scores are consistent with those of that scorer.
% ?
% The scorer preserves the settings used in the previous major shared tasks (e.g. {\CONLL}), and is faithful to the scorers developed in those shared tasks. 

Three score-reporting options are available:
%Therefore, there are three options for reporting those scores: 
%
The first option mirrors the evaluation used in the  {\CONLL} shared tasks \CITE{pradhan2012conllst} which excludes  singletons and split-antecedents from  evaluation. 
In this setting,  split-antecedents are ignored when constructing the clusters, and after the clusters are constructed singletons are %will be 
filtered out. 
I.e., only clusters with multiple mentions are evaluated. 

The second option is %the same as 
the one used in the identity anaphora sub-task of the {\CRAC} shared task \CITE{poesio-etal-2018-anaphora}. 
This evaluation includes  singletons, but not  split-antecedents. 
For this setting,  split-antecedents are ignored when constructing the clusters but  singletons are kept for the evaluation.

Finally, the scorer can  include both singletons and split-antecedent anaphors; this is the format used in  {\CODICRAC} \CITE{khosla-et-al:CODICRAC21}. 
Clusters include both %are constructed with  
split-antecedents and singletons. 
For split antecedents, a generalization of the existing coreference metrics was developed; 
this is briefly reviewed %we provide  a short overview of the approach 
in the next subsection.\footnote{Due to the complexity of the proposed method, 
we provide a full description 
%we describe the detailed approaches 
in a separate paper \CITE{paun-et-al:plural-scorer} which focuses on the evaluation of split-antecedent anaphora.}

\subsection{Split Antecedent Anaphora}

As discussed in Section \ref{sec:scoring}, 
the evaluation metrics for split antecedent anaphora proposed in 
%the solutions from 
previous work (e.g. \CITEA{vala-etal-2016-antecedents,zhou&choi:COLING2018}) are not entirely satisfactory. 
The {\UA} scorer implements a new method proposed by  \CITEA{paun-et-al:plural-scorer},
for scoring split-antecedent anaphora based on the idea of treating the antecedents of split-antecedent anaphors as a new type of mention, \NEWTERM{accommodated sets}--set denoting entities which have the split antecedents as elements. 
% \begin{exe}
% \ENEW{ex:split}
%     [John]$_1$ met [Mary]$_2$. [He]$_1$ greeted [her]$_2$. [They]$_{1,2}$ went to the movies.
% \end{exe}
%to treat  split-antecedents as special mentions that refer to multiple entities. 
So for instance, in example \SREF{ex:split}, split-antecedent anaphor $[They]_{1,2}$ is encoded as belonging to a coreference chain whose first element is the accommodated set \{1,2\} with the coreference chains for \LINGEX{John} and \LINGEX{Mary} as elements.
Schematically,

 \begin{center}
  %\small
     [He]$_1$ $\in$ Coref Chain 1 (\textbf{John}) 
           = \{ [John], [He] \}\\
     {[her]$_2$ $\in$ Coref Chain 2 (\textbf{Mary}) 
             = \{ [Mary], [her] \}} \\
     {[They]$_{1,2}$  $\in$ Coref Chain 3    
              (\textbf{\{John,Mary\}}) = 
              \{ \{1,2\}, [they] \}}
 \end{center}
The proposed generalization also gives
%We also introduce 
partial credit to interpretations of split-antecedent anaphors which do not identity all split antecedents.
% scoring to score split-antecedents by the similarity between a pair of split-antecedents in key and response. 

More specifically, to include  split-antecedents in the evaluation, the scorer first identifies all  accommodated sets %split-antecedents 
in the key and response. 
The relevant F1 scores are then calculated for pairs (key-response) of 
accommodated sets %split-antecedents 
to create a similarity matrix between all accommodated sets %split-antecedents 
in the key and response. 
The Kuhn-Munkres algorithm \CITE{kuhn1955hungarian,munkres1957algorithms} is used to search for the best alignments. 
To preserve the uniqueness of the different metrics, the relevant F1 scores are computed by applying the metric in question to the singular clusters that form the split-antecedents. 
For example, when computing the {\MUC} score, the {\MUC} F1 is used to find the optimal alignment between  accommodated sets %split-antecedents 
in the key and the response. 
Once the accommodated sets %split-antecedents 
are aligned, the standard metrics are adjusted to allow  partially matched mentions 
(i.e. the accommodated sets). %split-antecedents 
The procedures for the standard mentions are unchanged, while for computation associated with accommodated sets, %split-antecedents, 
partial credits are rewarded on how well the accommodated sets are %split-antecedents is 
resolved. 
The treatment for individual metrics are slightly different depending on the nature of the metrics.
We refer the reader to \CITEA{paun-et-al:plural-scorer} for  detailed discussion.

The scorer also provides an additional option to allow computing scores for 
split-antecedent plurals 
only.
This option is useful to assess a system's performance on resolving  split-antecedent references, which are not very frequent.
The score is defined as the micro-average F1 of all the split-antecedent anaphors in the key and response for all the supported metrics. 
% This option provides a useful tool to assess the quality of a system on resolving the split-antecedent references.

\subsection{Non-referring expressions}

A key aspect of anaphoric interpretation is correctly determining whether nominal phrases  like markable \LINGEX{it}  in \SREF{ex:non-referring} are   referring or not, and to distinguish such noun phrases from singleton mentions.

\begin{exe}
\ENEW{ex:non-referring}
    [It] was late at night.
\end{exe}
In the {\CONLLUA} exploded format,
non-referring expressions are associated with pseudo entities 
%in the `exploded' format 
%and are specified 
in the {\CODE{Identity}} column-- 
i.e., a `\CODE{-Pseudo}' suffix is appended to the cluster ids (\CODE{EntityID}) to distinguish them from  singleton mentions. 
The semantic type (\CODE{SemType}) attribute is used to specify the non-referring type in detail for corpora such as {\ARRAU} or  {\CODICRAC} in which such distinctions are made  (e.g. predicate, idiom). 
The following is an example of how a non-referring (predicative) NP is specified in the \CODE{Identity} column:

{
\begin{verbatim}
(EntityID=4-Pseudo|\ 
MarkableID=markable_6|\ 
Min=17|\ 
SemType=predicate)
\end{verbatim}
}

%To avoid duplication in the evaluation, 
The new {\UA} scorer follows the scorer developed for the {\CRAC} 2018 Shared Task in that 
non-referring expressions 
are not %will not be 
treated as singletons in the evaluation of  
identity reference. 
Instead, non-referring expressions are separated from  identity references when inputted to the scorer. 
More specifically, the collection of non-referring expressions in both the key and the response is identified and the scorer computes an F1 score 
for non-referring expressions only.
%between the two collections of non-referring expressions. 
The F1 score for non-referring expression is reported separately from the F1 scores for identity reference.


\subsection{Discourse Deixis}
% \todo{This is actually one of the key novel ideas of this paper, emphasize}
The {\UA} scorer supports the extension to discourse deixis proposed in version 1.0 of the Universal Anaphora specification of anaphoric phenomena by  implementing an entirely new approach to evaluation of discourse deixis supporting the evaluation.
This new approach is enabled by the way discourse deixis is encoded in the {\UA} markup. 

As mentioned in Section \ref{sec:scoring}, 
in the most recent previous work %previously 
%the 
discourse deixis is evaluated using the `Success@N' metric \CITE{kolhatkar&hirst:EMNLP14}, which is based on the assumption that  gold anaphors are given. 
The metric gives credit to a system if the gold segments are retrieved within the top N interpretations of the system. 
This approach is limited, both in that it requires gold anaphors, and because as it treats individual anaphors separately  without assessing the quality of the clusters they formed. 



% As mentioned in Section \ref{sec:scoring}, 
% in the most recent previous work %previously 
% %the 
% discourse deixis was evaluated using the `Success@N' metric \CITE{kolhatkar&hirst:EMNLP14}, which is based on the assumption that  gold anaphors are given. 
% The metric gives credit to a system if the gold segments are retrieved within the top N interpretations of the system. This approach is limited, as it treats individual anaphors separately and without assessing the quality of the clusters they formed. 

But discourse deixis is similar to  coreference, in that both form clusters by linking the anaphors to their antecedents.
Another important similarity is that in both cases we can have split-antecedent anaphors that refer to multiple antecedents--in fact, split antecedent reference is the norm for discourse deixis.
The main difference is that, in coreference,  antecedents are introduced using nominal phrases, whereas  in discourse deixis they are introduced using non-nominal phrases (segments). 

In the {\UA} markup, discourse deixis is specified in the {\CODE{Discourse\_deixis}} column of the `exploded' format,
and the same attributes are used as for the \CODE{Identity}
column.
%and uses similar attributes as used for the identity references. 
The only %main 
difference is that the cluster id (\CODE{EntityID}) and the markable id (\CODE{MarkableID}) of the segments are highlighted with a `\CODE{-DD}' suffix and `\CODE{dd\_}' prefix respectively, 
to avoid confusion in visual inspection. 
%This distinguishes them from the identity references to avoid any confusion. 
An example  {\CODE{Discourse\_deixis}} row is: 

{
\begin{verbatim}
(EntityID=1-DD|\ 
MarkableID=dd_markable_2|\ 
Min=19,32|\ 
SemType=dn|\ 
ElementOf=6-DD)
\end{verbatim}
}
%the antecedents are segments. 
This representation enables 
%However, this does not prevent 
the application of coreference metrics to evaluate discourse deixis. 
Particularly given that %Especially 
our new scorer provides a way to incorporate split-antecedents into the standard metrics, which therefore are discourse deixis-ready. 
This is exactly how the {\UA} scorer evaluates discourse deixis: it computes the same {\MUC}, {\BCUBED}, {\CEAF}, {\CONLL}, {\BLANC} and {\LEA} metrics as for identity anaphora. 

In other words, the {\UA} scorer introduces two novelties in the scoring of discourse deixis. 
First, discourse deixis  is evaluated in the same way as entity anaphora; this makes it possible to use the same  metrics used for identity reference for evaluating  discourse deixis as well.
One of the advantages of this approach is that discourse deixis evaluation now works with  predicted mentions/segments, which we hope will encourage research on discourse deixis to move to a more realistic setting.
Second, by adopting the generalization of the standard identity reference metrics to split antecedents, 
we can use the scorer for the very common case of discourse deixis with more than one segment antecedent. 
% as a generalized case of event coreference, allowing for split antecedents.  Hence we use the same identity reference metrics for evaluating  discourse deixis as well. 





\subsection{Bridging References}

In  {\UA} format, 
bridging references are specified in the {\CODE{Bridging}} column of the `exploded' format. 
The attributes for bridging include the markable ID (\CODE{MarkableID}), 
a mention of anchor entity (\CODE{MentionAnchor}), 
the cluster id of the antecedent (\CODE{EntityAnchor}) and the bridging relationship (\CODE{Rel}). 
For example:
%of the bridging annotation: 


{\begin{verbatim}
(MarkableID=markable_9|\ 
Rel=subset-inv|\ 
MentionAnchor=markable_1|\ 
EntityAnchor=3)
\end{verbatim}}


For bridging references, the scorer reports  three scores:  the two metrics computed by the scorer used for {\CRAC} 2018 shared task--mention-based F1 and  entity-based F1--and, in addition,  anaphora recognition F1. 

Mention-based F1 for bridging evaluates a system's ability to predict the correct anaphora and the mention of the anchor specified in the annotation (this is usually the closest or most suitable mention). 
\CODE{MentionAnchor} is used for this type of evaluation. 

Entity-based F1 is more relaxed than  mention-based F1, and does not require the system to predict exactly the same mention as the gold annotation. 
Instead, a system's interpretation is deemed correct as long as any mention of the correct anchor (\CODE{EntityAnchor}) is found, as done e.g., in \CITEA{poesio-etal-2018-anaphora}. 

Finally, anaphora recognition F1 is used to assess the system's ability to identify  bridging anaphors.  
