\section{Introduction}
\label{sec:introduction}

%\todo{To be revised}

Although the scope and ambition of anaphoric annotation projects has enormously increased in the last twenty years \CITE{poesio:ACL-discanno04,%
hinrichs-et-al:TLT04,pradhan-et-al:ICSC07,%
pradhan2012conllst,poesio&artstein:LREC08,%
uryupina-et-al:NLEJ,recasens&marti:LRE10,%
rahman&ng:EMNLP2012,%
nedoluzhko:LAW13,muzerelle-et-al:LREC14,%
cohen(kevin)-et-al:BMC17:CRAFT,Zeldes2017,%
webster-et-al:TACL2018,bamman-et-al:LREC20,%
Sakaguchi-aaai-20-winogrande,%
khosla-et-al:CODICRAC21,yu-et-al:CODI-CRAC-22,%
nedoluzhko-et-al:LREC22:CorefUD} 
a number of open questions about anaphoric annotation remain, and many if not most of the existing corpora have limitations either in size or coverage. 


The {\ARRAU} annotation  \cite{poesio&artstein:LREC08,uryupina-et-al:NLEJ,poesio-etal-2018-anaphora} is a long-term project to expand the range of anaphoric annotation by creating an anaphorically annotated corpus covering a wide variety of aspects of anaphoric reference \cite{poesio:ana-book-linguistic}.
%in a variety of genres,  including both written text and spoken language. 
The annotation project started in 2004 as the result of a series of studies of the reliability of 'difficult' aspects of anaphoric annotation  \cite{poesio:ACL-discanno04,poesio&artstein:ACL-ANNO-05,poesio&artstein:CORPLING-05,artstein&poesio:BRANDIAL06,artstein&poesio:CL}
and the first release was primarily focused on anaphoric reference in dialogue \cite{poesio&artstein:LREC08}.
The scope of the annotation then broadened both  in terms of linguistic aspects that were annotated and in terms of genres,  resulting in a second release in 2013 \cite{uryupina-et-al:NLEJ}. 
This second release was then used as the core dataset for the 2018 {\CRAC} Shared Task \cite{poesio-etal-2018-anaphora}, the first shared task for anaphora resolution covering also identification of non-referring expressions, bridging reference and discourse deixis;  and as additional material for the 2021 and 2022 {\CODICRAC} shared tasks on anaphora resolution in dialogues \cite{khosla-et-al:CODICRAC21,yu-et-al:CODI-CRAC-22}. 
These shared tasks highlighted the need to revise the annotation guidelines for a range of phenomena including 
%in particular 
discourse deixis and genericity and 
%the annotation of 
reference in dialogues. 
They also revealed a number of issues with tokenization and markup. 
We therefore started an extensive 
%process of 
reannotation and cleaning up, resulting in a third, substantially revised  release of the corpus. 

In this paper we discuss the issues identified with the previous annotation, the revised annotation scheme and guidelines,  the cleaning up procedure, and the new corpus resulting from this effort.

% The performance of models for single-antecedent anaphora resolution on the aspects of anaphoric interpretation annotated in the 
% reference %standard 
% {\ONTONOTES} dataset \CITE{pradhan2012conllst}
% has greatly improved in recent years \CITE{wiseman2015learning,lee2017end,lee2018higher,kantor-globerson-2019-coreference,joshi2020spanbert}. 
% So %recently 
% the attention of the community has started to turn to 
% more complex 
% cases of anaphora not 
% represented %found or not properly tested 
% in {\ONTONOTES}.

% Well-known examples of this trend are 
% work on the cases of 
% anaphora  whose interpretation requires some form of commonsense knowledge tested by benchmarks for the Winograd Schema Challenge \CITE{rahman&ng:EMNLP2012,%peng-et-al:NAACL2015,
% liu-ijcal-17-winograd,Sakaguchi-aaai-20-winogrande},
% or the pronominal anaphors that cannot be resolved purely using gender, 
% for which  benchmarks such as \ACRO{gap} have been developed \CITE{webster-et-al:TACL2018}.
% Another fruitful line of research has been devoted to creating datasets covering genres other than news, such as 
% conversation \CITE{muzerelle-et-al:LREC14,uryupina-et-al:NLEJ,khosla-et-al:CODICRAC21}, 
% fiction \CITE{bamman-et-al:LREC20}
% or scientific articles \CITE{cohen(kevin)-et-al:BMC17:CRAFT}.
% % Many of these datasets, %\ACRO{gap}, 
% % however, still focuse on identity coreference. 


% Further %In addition, more 
% research has been carried out on aspects of anaphoric interpretation that go beyond identity anaphora but are  covered by datasets such as {\ARRAU} \CITE{poesio-etal-2018-anaphora,uryupina-et-al:NLEJ} and {\GUM} \CITE{Zeldes2017} for English, 
% the Prague Dependency Treebank \CITE{nedoluzhko:LAW13} for Czech, and {\ANCORA} for Catalan and Spanish \CITE{recasens&marti:LRE10}.
% These include, e.g., bridging reference \CITE{clark:bridging,hou-et-al:CL18,hou-2020-acl,yu-etal-2020-bridging,kobayashi-ng-2021-bridging},
% discourse deixis \CITE{webber:91,marasovic-etal-2017-mention,kolhatkar-et-al:CL18} 
% or split-antecedent anaphora  \CITE{eschenbach1989remarks,vala-etal-2016-antecedents,zhou&choi:COLING2018,yu-etal-2020-plural,yu2021together}.

% %In addition, 
% % There has been interest in other genres apart from news. 
% % This includes substantial research on annotating and resolving coreference in biomedical and other scientific domains \CITE{cohen(kevin)-et-al:BMC17:CRAFT,lu&poesio:CRAC21} 
% % as well as in literary documents \CITE{bamman-et-al:LREC20}. 
% % There are, however, language genres still understudied in the literature on anaphoric reference.
% % Arguably the most important among these is conversational language in dialogue.
% % Anaphora resolution in dialogue requires systems to handle 
% % grammatically incorrect language suffering from disfluencies. Dialogue involves much more deictic reference, vaguer anaphoric and discourse deictic reference,  speaker grounding of pronouns and long-distance conversation structure. These are complexities that are often missing in news or Wikipedia articles, which form a large chunk of current datasets for coreference resolution. 
% % There has been some research on coreference in dialogue \CITE{byron:ACL02,eckert&strube:01,mueller:thesis}, but very limited in scope (primarily related to pronominal interpretation), due to the lack of suitable corpora. 
% % The one language for which substantial corpora of coreference in dialogue exist is French: the \ACRO{ancor} corpus \CITE{muzerelle-et-al:LREC14} has enabled  the development of an  end-to-end neural model for coreference interpretation in dialogue by \CITEA{grobol:2020:thesis}.
% % For English, the one resource we are aware of fully annotated for anaphoric reference is the {\TRAINS} corpora included in the {\ARRAU} corpus \CITE{uryupina-et-al:NLEJ}.

% The objective of the Universal Anaphora initiative, or {\UA},\footnote{\url{http://www.universalanaphora.org}} is to coordinate these efforts to push forward the state of the art in anaphora research. 
% The initiative, modelled on  Universal Dependencies,\footnote{\url{https://universaldependencies.org/}}  aims to achieve this by 
% expanding the aspects of anaphoric interpretation
% %identifying the anaphoric information 
% which are or can be reliably annotated in  anaphoric corpora,
% producing unified standards to annotate and encode these annotations, 
% delivering datasets encoded according to these standards, 
% and developing methods for evaluating this type of interpretation. 

% Like Universal Dependencies, Universal Anaphora is meant to push forward the state of the art in anaphora both from the linguistic and from the {\NLP} perspective. 
% One key issue  in this last regard is how to assess a system's ability to carry out these more advanced types of anaphoric interpretation.
% This 
% requires a scorer that can evaluate the interpretation produced by a system for, e.g., bridging reference, discourse deixis, or split-antecedent plurals. 
% Partial scorers exist and have been used, e.g., in the 2018 {\CRAC} Shared Task \CITE{poesio-etal-2018-anaphora}.
% However, no standardized scorer exists for many of these aspects of anaphoric interpretation, and the partial scorers suffer from a number of limitations. 
% Also, the solutions proposed for some of these aspects of interpretation, such as split-antecedent anaphors \CITE{vala-etal-2016-antecedents,zhou&choi:COLING2018,yu-etal-2020-plural} are not entirely satisfactory, 
% % for the reasons discussed in detail in Section \ref{sec:relatedwork}, 
% even though these are cases of identity anaphora after all.
% %and should therefore be in the scope of the Reference Coreference Scorer \CITE{pradhan-etal-2014-scoring}.

% %The objective of 
% In this paper we present the new Universal Anaphora scorer for anaphoric interpretation, 
% a Python  extension of the Reference Coreference Scorer \CITE{pradhan-etal-2014-scoring} 
% and of the Generalized Coreference Scorer developed by Moosavi for the {\CRAC} 2018 Shared Task.\footnote{\url{https://github.com/ns-moosavi/coval}}
% The {\UA} scorer is the first scorer
% able to evaluate system performance in all  aspects of anaphoric interpretation covered by the current version of the Universal Anaphora proposal. 
% This scorer was used in the {\CODICRAC} Shared Task in Anaphora Resolution in Dialogue\footnote{\url{https://competitions.codalab.org/competitions/30312}} 
% % whose objective was to provide participants with the opportunity to develop automated approaches for coreference resolution that tackle less studied forms of anaphora and generalize on different types of conversational setups
% \CITE{khosla-et-al:CODICRAC21}
% and will be used in the forthcoming 2022 edition of the shared task.\footnote{\url{https://codalab.lisn.upsaclay.fr/competitions/614}}

% We begin with some background on 
% the Universal Anaphora initiative in Section~\ref{sec:UA}.
% %, as no paper discussing the initiative has been published yet. 
% Next, we discuss research on scoring anaphora in Section \ref{sec:scoring}. 
% The new scorer is presented in Section \ref{sec:scorer}.
% In Section \ref{sec:task}, we  discuss the {\CODICRAC} shared task.
% In Section \ref{sec:results}, we analyze its results, with a  focus on issues related to the scoring.
% %rather than system performance, which is covered in \CITE{khosla-et-al:CODICRAC21}.

% %\todo{To be completed}