\section{Correcting tokenization and conversion errors}
\label{sec:corrections}

{\ARRAU} 3 fixes a couple of errors and inconsistencies in  the markup in previous versions.
If the corrections resulted in modifications to the underlying text (the \emph{basedata} in {\MMAX}2 parlance), existing annotations were adapted such that they were still valid. 
Depending on the complexity of the corrections, they were performed in a fully or semi-automatic manner (based on scripts using \mbox{pyMMAX}2 \cite{mueller2020}), with manual checks afterwards. 

\subsection{Tokenization}
\label{sec:tokenization}
Tokenization, i.e.\ splitting of text into basedata elements, was improved for all sub-corpora by using a more fine-grained splitting scheme than the previous one, which was only sensitive to white space and punctuation.
%The method includes a basedata tokenizer which uses Unicode character categories for determining token boundaries in the input string in a universal, language-independent way. 
%This procedure is rather aggressive, creating considerably more and shorter tokens than ’standard’ tokenizers, which are often only sensitive to white space and punctuation. 
Most notably, basedata is split at word-internal non-word characters, including hyphens. 
%\todo{This goes beyond mere 'correction', so do we / you want that in ARRAU, at all? It might also have consequences for the existing annotations: the compound token 'asbestos-related' is split now, so it would be possible to annotate some relation from its modifier to some other occurrences of asbetos (if that would be correct, that is).}
As a result, hyphenated words (e.g.\ noun compounds and other hyphenated multi-word expressions) will be separated into several contiguous basedata elements, allowing for more fine-grained annotation. 
At the same time, tokenization keeps track of the original input string composition, including white space, and stores, for every basedata element, the number of leading white space characters.
This way, the original text appearance can be reproduced in the the annotation tool {\MMAX}2, allowing for a 
%much more appealing and 
better-to-read, more natural and less distracting rendering of the display.

\subsection{PRD Conversion Errors}
Some errors were found in the {\RST} portion of the dataset. 
The {\RST} portion was originally converted from the Penn Treebank \ACRO{prd} format. 
During the first round of checks, we discovered that this conversion had introduced a couple of errors.
{\NP}s for numbers which contained commas as separators (Example \SREF{ex:prd1}, from \CODE{WSJ\_0012}) were incorrectly truncated, resulting in only the first number (\emph{4} in the example) to be imported into the {\ARRAU} data.
\begin{exe}
\ENEW{ex:prd1}
\begin{verbatim}
 ...
 (NP (NP average circulation)
     (PP of
     (NP (NP 4,393,237))))
 ...
\end{verbatim}
\end{exe}
Sentence annotations, which are instrumental for structuring the annotation tool display by adding sentence-final line breaks, are derived from the \ACRO{prd}s top-level S-bracketings. 
In previous versions of {\ARRAU}, sentence annotations frequently left out trailing punctuations, causing both sentence-final markables to be incomplete, and the display to be incorrect.
%
Yet another class of errors in previous versions of {\ARRAU} were caused by imperfect creation of the \ACRO{prd} files from the original raw files, in cases where the original text contained slashes. Example \SREF{ex:prd2}, from \CODE{WSJ\_0207}, shows the rendering in the \ACRO{prd} file (which is also used in {\ARRAU}. In the original file, however, which is also distributed with {\ARRAU}, the actual text reads "11 1/2 minutes".

\begin{exe}
\ENEW{ex:prd2}
\begin{verbatim}
...
(VP lasts
 (NP-TMP (QP 11 1) minutes))
...
\end{verbatim}
\end{exe}
While none of these issues are critical, correcting them may also help a 
future integration 
%future possible linking integrating 
in the corpus 
of
other types of annotation available for the {\RST} subset, in particular discourse structure but also for instance PropBank information.