\section{ARRAU 1 and 2}
\label{sec:ARRAU2}

% Plan for this Section:

% \begin{itemize}
%     \item Quick description of {\ARRAU} 2 with discussion of some key aspects of the annotation scheme and some statistics
% \end{itemize}

% \paragraph{Genres}  The first release  of the {\ARRAU} corpus \cite{poesio&artstein:LREC08} focused on dialogue. 
% In particular, its
% {\TRAINS} domain of task-oriented dialogues includes
% a complete annotation of the {\TRAINS-93}  
% corpus\footnote{\url{http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC95S25}} 
% and the pilot dialogues in the so-called {\TRAINS-91} corpus.

% The second release of the {\ARRAU} corpus\footnote{\url{http://www.arrauproject.org}} \cite{uryupina-et-al:NLEJ}
% was designed to cover a broader variety of genres than the first.
% Version 2 includes a substantial amount of news 
% text in a  
% sub-corpus called \CODE{RST}, consisting of the  Penn Treebank \cite{marcus-et-al:CL93}.
% In addition,  {\ARRAU} Version 2 includes a complete annotation of the spoken narratives in the Pear Stories 
% \cite{chafe:80},
% and  documents in the medical and art history genres from the {\GNOME} corpus \cite{poesio:ACL-discanno04}.

\subsection{Genres} 

The {\ARRAU} corpus\footnote{\url{http://www.arrauproject.org/corpus}} \cite{poesio&artstein:LREC08,uryupina-et-al:NLEJ}
was designed to cover a variety of genres. 
Initially, the corpus was 
meant to focus on %created to cover 
anaphoric reference in dialogue and spoken language \cite{poesio&artstein:LREC08}.
Its \CODE{TRAINS} sub-corpus includes all the task-oriented dialogues in the {\TRAINS-93}  
corpus\footnote{\url{http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC95S25}} \cite{heeman&allen:trains-corpus-93}
already used in Byron's work on  pronominal reference in dialogue \cite{byron&allen:98,byron:ACL02}
as well as the pilot dialogues in the so-called {\TRAINS-91} corpus. 
The \CODE{PEAR} sub-corpus consists of 
the complete collection of spoken narratives in the Pear Stories that provided some of the early evidence on salience and anaphoric reference   
\CITE{chafe:80}.\footnote{\url{https://www.linguistics.ucsb.edu/research/pear-film}}
Subsequently, the corpus was extended to 
cover a substantial amount of written text, including news text in a  
sub-corpus called {\CODE{RST}}, consisting of the entire subset of the Penn Treebank \CITE{marcus-et-al:CL93}
that was annotated in  the \ACRO{rst} treebank
\CITE{carlson(lynn)-et-al:03}.\footnote{\url{https://catalog.ldc.upenn.edu/LDC2002T07}}
The \CODE{GNOME} sub-corpus covers documents from the medical and art history genres covered by the {\GNOME} corpus \CITE{poesio:ACL-discanno04}.





\subsection{Annotation scheme} 

The same coding scheme was used for all sub-corpora, 
but separate guidelines were written for the 
spoken dialogue and written language sub-corpora.
The original annotation scheme used for Release 1 \cite{poesio&artstein:LREC08}, focused on dialogue, is distributed with the dataset and is also available  from the {\ARRAU} corpus page.
%
For the second release \cite{uryupina-et-al:NLEJ}, the guidelines for bridging were extended and genericity was also annotated using the {\GNOME} guidelines, but a complete new manual was not produced. However, a fairly extensive description can be found in \CITEA{uryupina-et-al:NLEJ}. 


\paragraph{Markable definition}
Many
older %, especially among the older, 
anaphorically annotated  corpora impose syntactic, semantic or discourse-based restrictions on
markables.
For instance, in {\ONTONOTES} neither expletives nor singletons
are annotated 
\CITE{poesio-et-al:ana-book-corpora}.
%(for a discussion of the state of the art in anaphoric annotation, see \CITEA{poesio-et-al:ana-book-corpora}).
By contrast,
in {\ARRAU} \textit{all} {\NP}s are considered as markables, including 
%when they are
non-referring expressions (e.g., %because either 
expletives such as \LINGEX{it}
or  predicative {\NP}s such as
\LINGEX{a busy place}) in
\SREF{ex:non-referring},
%\LINGEX{[It] seems to be [a busy place]}, 
%and when they 
and expressions do not corefer with any other markable 
(`singletons'). 
%and thus form a singleton coreference chain.
Moreover, in {\ARRAU} non-referring markables are manually sub-classified into
expletives, predicative, and quantifiers. % (see below).
In addition, all generic references are marked, including
premodifiers when the  entity referred to
is mentioned again, e.g., in the case of the proper name \LINGEX{US} in
\SREF{ex:ARRAU-premodifiers:a}, 
and premodifiers that refer to a  kind, like
\LINGEX{exchange-rate} in \SREF{ex:ARRAU-premodifiers:b}. 

\begin{exe}
% \ENEW{ex:non-referring} 
%       [It] seems to be [a busy place]
\ENEW{ex:ARRAU-premodifiers:a}
\ldots The Treasury Department said that the [US]_1 trade deficit may worsen next year after two years of significant improvement\ldots
The statement was the [US]_1's government first acknowledgment 
\ldots
%of what other groups, such as the International Monetary Fund, have been predicting for months. 

\ENEW{ex:ARRAU-premodifiers:b}
The Treasury report, which is required annually by a provision
of the 1988 trade act, again took South Korea to task for its
[exchange-rate]_1 policies. ``We believe there have continued to be
indications of [exchange-rate]_1 manipulation \ldots
\end{exe}
A distinctive feature of {\ARRAU}'s definition of markables is that, due to its initial focus on dialogue, it also allows \NEWTERM{discontinuous} markables such as the collaborative constructed \LINGEX{three ... loaded boxcars} in \SREF{ex:discontinuous}, building on
%M{\"{u}}ller's work 
\cite{mueller:thesis} and leveraging {\MMAX}2's support for such markables.

\begin{exe}
    \ENEW{ex:discontinuous}
    \begin{tabular}{cl}
      S:  &  okay um if you can only pull three  \\
         &  loaded boxcars\\
      U:  & {[three]$^1$}\\
      S:  & yeah [loaded boxcars]$^1$
    \end{tabular}
\end{exe}

% \paragraph{Genericity}

% {\ARRAU} is not a multi-layer corpus like other  corpora such as {\ANCORA}, {\GUM}, {\ONTONOTES} or the Prague Dependency Treebank, meaning that all the information relevant for the study of anaphora (morphosyntax, dependency structure, semantics) has to be annotated within the anaphoric layer.
% We only discuss in this paper the \CODE{generic} attribute, as it's the one of these attributes for which the guidelines changed in {\ARRAU} 3.

% The {\ARRAU} scheme and guidelines for genericity build on the studies of genericity reliability carried out as part of the {\GNOME} annotation \cite{poesio-et-al:ACL04}.
% This scheme is based on a generalised notion of scopal dependence for nominals covering both genericity and scopal dependence on a range of operators including conditionals, quantifiers, and temporal adverbials. 

% More specifically, 
% according to the guidelines used for 
%  {\ARRAU} 1 and 2,
% the annotation of the \CODE{generic} attribute is carried out following a decision tree going from the easiest cases to the more complex ones.
% Coders are first asked to check whether the nominal  is  in the syntactic scope of an \emph{explicit} operator such as 
% a conditional  like \LINGEX{if} (as in \SREF{ex:genericity:op-conditional}).
% or an  individual quantifier such as \LINGEX{every} or \LINGEX{most} (\CODE{iquant}) 
% %(as in \SREF{ex:genericity:iquant}).
% % or a temporal quantifier like \LINGEX{always} or \LINGEX{once} (as in \SREF{ex:genericity:tquant})
% % a modal (as in \SREF{ex:genericity:modal}) 
% %or an instruction (as in \SREF{ex:genericity:instruction}). 
% In these cases, the nominal is \textit{not} marked as generic, but as being in the scope of the appropriate operator. 
% If no such explicit quantifier/operator is present, coders are asked to check 
% whether the nominal refers to  semantic objects whose genericity is  left underspecified, such as substances (e.g., \textit{gold}), as in 
% \SREF{ex:genericity:underspecified-substance:RST} 
% % seen before or in 
% % \SREF{ex:genericity:underspecified-substance:GNOME}.
% Finally, the annotator is asked whether the sentence in which the markable occurs is generic, 
% and in this case, 
% to mark the nominal as  \CODE{generic-yes} if it refers generically, as in \SREF{ex:genericity:generic-yes},
% or \CODE{generic-no} otherwise.
% With these instructions, reasonable intercoder agreement was  achieved 
% ($\kappa = .82$) \cite{poesio:ACL-discanno04}.

% \begin{exe}
% \ENEW{ex:genericity:op-conditional}
% New York State Comptroller Edward Regan predicts a \$ 1.3 billion budget gap for the city `s next fiscal year, a gap that could grow if there is $[$a recession$]^{operator-conditional}$.`` %\\ 
% %(\CODE{operator-conditional})
% % \item  \label{ex:genericity:op-conditional}
% % S: okay so that was so we'll get to Dansville about four AM now the other problem is is that if we have $[$this train$]$ so we also have engine E one that's going back and forth right $[$operator-conditional$]$
% %\ENEW{ex:genericity:iquant}
% %Mr. Uhr said that Mr. Petrie or his company have been accumulating Deb Shops stock for several years, each time issuing $[$a similar regulatory statement$]^{operator-iquant}$. %\\
% %(\CODE{operator-iquant})
% % \ENEW{ex:genericity:tquant}
% % In addition , once $[$money$]^{generic}$ is raised , $[$investors$]^{generic}$ usually have no way of knowing how $[$it$]^{generic}$ is spent.\\ 
% % (\CODE{operator-tquant})
% % %NB: coreference for generic here
% % \ENEW{ex:genericity:modal}
% % They argue that their own languages should have $[$equal weight$]^{generic}$, although recent surveys indicate that the majority of the country`s population understands Filipino more than any other language.\\
% % (\CODE{operator-modal})
% % \ENEW{ex:genericity:instruction}
% % Use $[$alcohol wipes$]^{generic}$ to clean the tops of the vials move in one direction and use one wipe per vial.\\ 
% % (\CODE{operator-instruction})
% \ENEW{ex:genericity:underspecified-substance:RST}
% Not that $[$oil$]^{undersp-substance}$ suddenly is a sure thing again .%\\
% %(\CODE{undersp-substance}, \CODE{RST})
% % \ENEW{ex:genericity:underspecified-substance:GNOME}
% % 1 ml of $[$the prepared solution for injection$]^{generic}$ contains 0.25 mg ( 8 million IU ) of $[$Interferon beta-1b$]^{generic}$.\\
% % \CODE{underspecified-substance}, \CODE{GNOME})
% \ENEW{ex:genericity:generic-yes}
% In its report to Congress on $[$international economic policies$]^{generic-yes}$, the Treasury said that any improvement in the broadest measures of trade, known as the  current account.%\\
% %(\CODE{generic-yes})
% %-- is the firs one (the prepared solution for injection) really generic? $]$}
% %\item  \ToDo{$[$generic-no$]$}
% % \item  okay $[$it$]$ takes an hour to load each boxcar
% %  \ToDo{$[$episodic-no. how is it different from expletive?$]$}
% %\item  \ToDo{$[$no-generic$]$}
% %\item  \ToDo{$[$operator-question$]$}
% %\item  \ToDo{$[$underspecified-decease$]$}
% %\item  \ToDo{$[$underspecified-generic$]$}
% %\item $[$one of them$]$ has a what do you call those little um paddleball
% %\ToDo{$[$underspecified-replicable$]$}
% \end{exe}

% % The version used in {\ARRAU} 1 and 2 ... 

\paragraph{Referential status}

% The {\ARRAU} guidelines require annotating a markable's referential status.
% %using the \CODE{reference} attribute. 
A markable can be marked as 
%being 
semantically  non-referring (an expletive, a predicate, a quantifier, a coordination, an idiom, or incomplete)
or referring (either \CODE{discourse new} or
\CODE{discourse old}). 
Discourse new mentions introduce new entities and thus are not marked as being coreferent with an entity already introduced  (\NEWTERM{antecedent}).
For discourse-old markables, the annotation of different types  of anaphoric relations is supported.
% All referring markables are marked as either \CODE{discourse new} or
% \CODE{discourse old}. 
The  antecedent of discourse-old mentions can be either of type
\CODE{phrase} (if the antecedent was introduced using a nominal markable) or  
\CODE{segment} (not introduced by a nominal markable,
for \NEWTERM{discourse deixis}).\footnote{Identity
  anaphora also includes split antecedent plural anaphoric reference.}   
In addition, referring {\NP}s can be marked as \NEWTERM{related} to a
previously mentioned discourse entity %in order
to identify them as
examples of  associative (\NEWTERM{bridging}) anaphora.

\paragraph{Bridging references}
Annotating --- indeed, even identifying --- bridging references in a reliable
way is difficult, which is one
of the reasons why so few large-scale corpora for anaphora include
this type of annotation \CITE{poesio-et-al:ana-book-corpora,kobayashi-ng-2020-bridging}. 
The {\ARRAU} guidelines for bridging anaphora are based on 
experiments that ran from 
%were started by Vieira and Poesio
\CITE{poesio&vieira:CL} 
to %and continued in the {\GNOME} project 
\CITE{poesio:ACL-discanno04}.
The {\ARRAU} Release 1 and 2 guidelines followed the {\GNOME} guidelines,
but with an extension and a simplification.  
Annotators were asked to mark a markable as \CODE{related} to a particular antecedent if it stood to that antecedent in one of the {\GNOME} relations 
%identified in  {\GNOME}
%(indeed, the same examples were used), 
or %and in addition, 
%if they stood 
in the two additional relations %(but
%without testing the reliability of this annotation): 
\begin{itemize}
\item \CODE{other}, for \LINGEX{other} {\NP}s, broadly following the
  guidelines in \CITEA{modjeska:thesis}; 
\item an \CODE{undersp-rel} relation for `obvious cases of bridging
  that didn't fit any other category'.  
\end{itemize}
% Some statistics about bridging references in {\ARRAU} Release 2 are
% shown in Table \ref{t:bridging}. 
However, the actual relations were not marked in {\ARRAU} 1. Relation annotation started with {\ARRAU} 2, but only for the {\CODE{RST}} %{\RST} 
portion. 
One of the objectives 
for {\ARRAU} 3 %of the {\ARRAU} 3 annotation 
was to annotate the relations  underlying bridging reference for all sub-corpora.

\paragraph{Discourse deixis}
Discourse deixis in its full form 
is a very complex form of reference,
both to annotate 
and to resolve \cite{kolhatkar-et-al:CL18} .
Very few anaphoric annotation projects have attempted to annotate
discourse deixis in its entirety
\CITE{kolhatkar-et-al:CL18}.
More typical is a partial annotation, as in
\CITE{byron&allen:98,navarretta:00}, 
%\todo{citation not added to bib file}
who annotated pronominal reference to abstract objects; 
in \ACRO{OntoNotes}, where event anaphora was marked
\CITE{pradhan-et-al:ICSC07}; 
and in %the work of 
\CITE{kolhatkar&hirst:EMNLP14}, which focused on
so-called shell nouns.
In {\ARRAU},
a coder  specifying that a referring expression is discourse-old
  is asked  whether its antecedent was introduced using a
  \CODE{phrase} (markable)  or a \CODE{segment} (discourse segment).   
Coders who choose \CODE{segment} %as the type of antecedent
  have to
  mark a sequence of \textit{predefined} clauses as antecedent.

\paragraph{Genericity}

{\ARRAU} is not a multi-layer corpus like 
%other  corpora such as 
{\ANCORA}, {\GUM}, {\ONTONOTES} or the Prague Dependency Treebank, meaning that 
%%in addition to information about anaphora, 
other linguistic information relevant for the study of anaphora (morphosyntax, dependency structure, semantics) also has to be annotated within the anaphoric layer.
We only discuss in this paper 
genericity, %the \CODE{generic} attribute, 
as it's the one among these attributes for which the guidelines changed in {\ARRAU} 3.

The {\ARRAU} scheme and guidelines for genericity build on the studies of genericity reliability carried out as part of the {\GNOME} annotation \cite{poesio-et-al:ACL04}.
This scheme is based on a generalised notion of scopal dependence for nominals covering both genericity and scopal dependence on a range of operators including conditionals, quantifiers, and temporal adverbials. 
%
More specifically, 
according to the guidelines used for 
 {\ARRAU} 1 and 2,
the annotation of the \CODE{generic} attribute is carried out following a decision tree going from the easiest cases to the more complex ones.
Coders are first asked to check whether the nominal  is  in the syntactic scope of an \emph{explicit} operator such as 
a conditional  like \LINGEX{if} (as in \SREF{ex:genericity:op-conditional})
or an  individual quantifier such as \LINGEX{every} or \LINGEX{most} (\CODE{iquant}) 
%(as in \SREF{ex:genericity:iquant}).
% or a temporal quantifier like \LINGEX{always} or \LINGEX{once} (as in \SREF{ex:genericity:tquant})
% a modal (as in \SREF{ex:genericity:modal}) 
%or an instruction (as in \SREF{ex:genericity:instruction}). 
In these cases, the nominal is \textit{not} marked as generic, but as being in the scope of the appropriate operator. 
If no such explicit quantifier/operator is present, coders are asked to check 
whether the nominal refers to  semantic objects whose genericity is  left underspecified, such as substances (e.g., \textit{gold}), as in 
\SREF{ex:genericity:underspecified-substance:RST} 
% seen before or in 
% \SREF{ex:genericity:underspecified-substance:GNOME}.
Finally, the annotator is asked whether the sentence in which the markable occurs is generic, 
and in this case, 
to mark the nominal as  \CODE{generic-yes} if it refers generically, as in \SREF{ex:genericity:generic-yes},
or \CODE{generic-no} otherwise.
With these instructions, reasonable intercoder agreement was  achieved 
($\kappa = .82$) \cite{poesio:ACL-discanno04}.

\begin{exe}
\ENEW{ex:genericity:op-conditional}
New York State Comptroller Edward Regan predicts a \$ 1.3 billion budget gap for the city `s next fiscal year, a gap that could grow if there is $[$a recession$]^{operator-conditional}$.`` %\\ 
%(\CODE{operator-conditional})
% \item  \label{ex:genericity:op-conditional}
% S: okay so that was so we'll get to Dansville about four AM now the other problem is is that if we have $[$this train$]$ so we also have engine E one that's going back and forth right $[$operator-conditional$]$
%\ENEW{ex:genericity:iquant}
%Mr. Uhr said that Mr. Petrie or his company have been accumulating Deb Shops stock for several years, each time issuing $[$a similar regulatory statement$]^{operator-iquant}$. %\\
%(\CODE{operator-iquant})
% \ENEW{ex:genericity:tquant}
% In addition , once $[$money$]^{generic}$ is raised , $[$investors$]^{generic}$ usually have no way of knowing how $[$it$]^{generic}$ is spent.\\ 
% (\CODE{operator-tquant})
% %NB: coreference for generic here
% \ENEW{ex:genericity:modal}
% They argue that their own languages should have $[$equal weight$]^{generic}$, although recent surveys indicate that the majority of the country`s population understands Filipino more than any other language.\\
% (\CODE{operator-modal})
% \ENEW{ex:genericity:instruction}
% Use $[$alcohol wipes$]^{generic}$ to clean the tops of the vials move in one direction and use one wipe per vial.\\ 
% (\CODE{operator-instruction})
\ENEW{ex:genericity:underspecified-substance:RST}
Not that $[$oil$]^{undersp-substance}$ suddenly is a sure thing again .%\\
%(\CODE{undersp-substance}, \CODE{RST})
% \ENEW{ex:genericity:underspecified-substance:GNOME}
% 1 ml of $[$the prepared solution for injection$]^{generic}$ contains 0.25 mg ( 8 million IU ) of $[$Interferon beta-1b$]^{generic}$.\\
% \CODE{underspecified-substance}, \CODE{GNOME})
\ENEW{ex:genericity:generic-yes}
In its report to Congress on $[$international economic policies$]^{generic-yes}$, the Treasury said that any improvement in the broadest measures of trade, known as the  current account.%\\
%(\CODE{generic-yes})
%-- is the firs one (the prepared solution for injection) really generic? $]$}
%\item  \ToDo{$[$generic-no$]$}
% \item  okay $[$it$]$ takes an hour to load each boxcar
%  \ToDo{$[$episodic-no. how is it different from expletive?$]$}
%\item  \ToDo{$[$no-generic$]$}
%\item  \ToDo{$[$operator-question$]$}
%\item  \ToDo{$[$underspecified-decease$]$}
%\item  \ToDo{$[$underspecified-generic$]$}
%\item $[$one of them$]$ has a what do you call those little um paddleball
%\ToDo{$[$underspecified-replicable$]$}
\end{exe}

% The version used in {\ARRAU} 1 and 2 ... 

\todo{Mention here the issues with dialogue annotation, in particular not marking first and second person pronouns in {\TRAINS}?}

\todo{Mention here the marking of ambiguity?}

\subsection{Annotation procedure}

{\ARRAU} 1 and 2 were annotated  using  {\MMAX}2  \CITE{mueller&strube:06}. 
All annotation was carried out by trained (computational) linguists.
{\ARRAU} 1 was primarily annotated at the University of Essex between 2004 and 2007 
% CUT TO SAVE SPACE
%by Laura Bostock, Rhiannon Jones and Tara Sutton, 
under the direction of Ron Artstein, who also designed the {\MMAX}2 style, and in  collaboration with Mark-Christoph M\"{u}ller. 
The initial annotation was then extended and checked as part of the Johns Hopkins 2007 Workshop on Entity Disambiguation (\ACRO{elerfed}).
% CUT TO SAVE SPACE
%by Ron Artstein, Janet Hitzeman, Mijail Kabadjov,  Massimo Poesio, and Josef Steinberger. 
%The annotation started with part of the {\TRAINS} 1991 and 1993 datasets, and was then extended to include a subset of the {\GNOME} corpus, the Pear Stories, and part of the {\RST} Discourse Treeebank already partially annotated by Andrej Kibrik's team at the Russian Academy of Sciences. 

{\ARRAU} 2 was %primarily 
annotated at the University of Trento between 2008 and 2016 under the coordination of Kepa Rodriguez, Francesca Delogu,  Federica Cavicchio, and Olga Uryupina. Most of the annotation was carried out by Antonella Bristot.
%a trained linguist.

\subsection{Use in shared tasks}

In recent years, {\ARRAU} was used for three shared tasks: the {\CRAC} 2018 shared task on anaphora resolution with the {\ARRAU} corpus \cite{poesio-etal-2018-anaphora}, and the 2021 and 2022 {\CODICRAC}
shared tasks on anaphora resolution in dialogue \cite{khosla-et-al:CODICRAC21,yu-et-al:CODI-CRAC-22}.

The use of the corpus for such tasks was enabled by two improvements brought about by the Universal Anaphora initiative.\footnote{http://www.universalanaphora.org}
The first of these was the development of a tabular markup format extending the {\CONLL}-U tabular format used for the {\CONLL} 2011 and 2012 shared tasks on coreference \cite{pradhan2012conllst} with ways to represent the additional types of anaphoric information encoded in {\ARRAU}, but consistent with it so that modellers would understand it better. 
And second, the development of scorers extending the Coreference Reference scorer \cite{pradhan-etal-2014-scoring} with ways of scoring the interpretation of these additional phenomena \cite{poesio-etal-2018-anaphora,yu-et-al:UA-scorer-22}.

%The important point of these shared tasks for the present paper is that they revealed a number of issues with {\ARRAU} 2, which wee aimed to address in the present release.

%  \CODE{RST}, \CODE{TRAINS}-93 and \CODE{PEAR} were used for the {\CRAC} 2018 shared task.