\documentclass[a4paper]{article}
\title{More Details about The Annotation System}
\author{ Zhenghua Li 
%Zhenghua Li, Yue Zhang\\
%Soochow Univeristy, Suzhou, China \\
%\tt{zhli13@suda.edu.cn} 
}
\date{February 20, 2018}
%\date{\today}

\setlength{\topmargin}{-10mm}
\setlength{\textwidth}{7in}
\setlength{\oddsidemargin}{-8mm}
\setlength{\textheight}{9.1in}
\setlength{\footskip}{1in}

\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{multirow}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage[dvips]{graphicx}
\usepackage[usenames]{color}
\usepackage{tikz-dependency}
\usepackage{subfigure}
\usepackage{url}

\graphicspath{{image/}}

\begin{document}
\fontsize{12}{15}
\selectfont
\maketitle

\section{Annotation Interface}

This is a brief introduction to our browser-based annotation platform, 
which we have built to better support our long-term data annotation and guarantee annotation quality. % (tested with Chrome and Sougou).
You can find the URL of our annotation system at \url{http://hlt.suda.edu.cn/~zhli/}. 
%The web page is \url{http://url}.
%\url{http://hlt-service.suda.edu.cn/syn-dep-batch}. Please try.


The annotation interface is shown in Figure \ref{fig:anno-interface}. 
To highlight selected words to be annotated (supporting both partial and full annotation), the system marks the corresponding words with red circles.
Annotators can only annotate head words for the selected words, and must complete all selected tasks before submission. 

\begin{figure}[hptb]
\centering
\includegraphics[width=1.0\textwidth]{anno-sys-interface}
\caption{
Annotation interface. Before annotation, the words to be annotated are marked with red circles; 
after annotation, the words are annotated with their heads and red circles disappear.
}
\label{fig:anno-interface}
\end{figure}


To annotate a dependency, annotators just need to first click the head word, and then click the modifier word, and finally select the relation label. 
Once the head word is decided, the corresponding red circle disappear.
Annotator may also cancel a dependency by right-click on the modifier word.

The annotated result, before submission, should be a legal tree or a part of a legal tree: 1) single-headed, 2) acyclic, 3) single-root. 


\section{Workflow Control}


We design a sophisticated workflow so that the data quality can be guaranteed through the platform and avoid further offline process of the annotated data, as shown in Figure \ref{fig:workflow}.


\begin{figure}[hptb]
\centering
\includegraphics[width=1\textwidth]{anno-workflow-v2}
\caption{
The workflow for annotating a sentence. 
}
\label{fig:workflow}
\end{figure}

First, the system dynamically assigns each task to two random annotators. 
If the two submissions are not the same (inconsistent dependency or relation label), a third expert annotator will compare them and decide a single answer. %Our annotation platform also provides 
%As a kind of feedback, % function. For example, 
%As a strategy of improvement
As a kind of feedback, the platform will return the answer to the annotator who makes mistakes and 
the annotator must correct his/her wrong submission accordingly for the sake of improvement. %  to the corresponding correct answers. 
If an annotator insists that the answer is wrong, he/she may click a complaint button %if the expert annotator is wrong, 
and the platform then distributes the task to a final senior expert. 
%In a word, we design a sophisticated procedure to guarantee the data quality. 
We find this procedure is extremely effective for %automatic 
quality control. 




\end{document}


While annotating one sentence, annotators just need to choose the head of each task in each sentence selected by a parser model following some uncertainty metric. To highlight tasks, the system marks tasks with red circles. To annotate one task, annotators click the head of one task firstly and then click the task. To highlight the click of head, the system shows a light gray shadow under the head. After clicks of head and task by order, the system hides the shadow of the head and the red circle of the task, and generates an arc from the head to the task with an arrow. If annotators realize their mistake click after the first click, they can click the head again and its light gray shadow will disappear. If annotators realize their mistake after the click of task after the arc has been generated, they can right click the task and the arc will disappear and the red circle appears again. The disappearance of all red circles means that all tasks have got their heads.

Annotators can click the submit button to submit their annotation results after they annotated all tasks. To get valid annotation result, the annotation system may check the results before saving them to database. Validity check only includes basic principles of the dependency parser, for example, all arcs cannot cross. If result violates any principle, the system refuses the submitting request from annotators and warns them with some information. Annotators can change their results according to the warning information before they submit their results successfully.

