\section{Evaluation Metrics}
\label{sec:evaluation}

% Martin: Následující věta se v podstatě opakuje v následujícím odstavci Official scorer.
%We evaluate all shared task submissions with the CorefUD scorer.
As in prior editions, the primary evaluation score is the CoNLL F$_1$ score, computed using head mention matching with singletons excluded, and a dependency-based procedure to match zero mentions.
Alongside the primary score, we report additional metrics to enable a broader comparison of submitted systems.

\paragraph{Official scorer}
Outputs of participating systems are scored with the CorefUD scorer,%
\footnote{\url{https://github.com/ufal/corefud-scorer}}
which has not been changed since the previous editions.
The scorer is based on the Universal Anaphora (UA) scorer 2.0 \citep{ua-scorer-2.0},
%\footnote{%
%The UA scorer 2.0 merges, reimplements, and extends several earlier tools, including previous versions of the CorefUD scorer.
%}
and inherits all functionality needed for the shared task, including implementations of standard coreference evaluation metrics.
Compared to the UA scorer, the CorefUD scorer additionally provides head matching and a dependency-based alignment method for zero mentions.

The scorer expects two CoNLL-U files: a gold file and a system prediction.
Because both our plaintext and JSON formats omit some information required for evaluation (e.g., mention heads), any output produced in these formats must be reconstructed into CoNLL-U prior to evaluation.

\paragraph{Mention matching}
Given the shortcomings of \emph{exact} and \emph{partial} mention matching (see \citet{oursharedtask2023} for details), we use \emph{head match} strategy for the primary metrics.
Under this criterion, a gold mention and a predicted mention are matched when they share the same head token.%
\footnote{In CorefUD, gold mention heads are derived from the dependency tree using the Udapi block \texttt{corefud.MoveHead}.}
Mention spans are otherwise disregarded, except when multiple mentions have an identical head, in which case spans are used to distinguish them.

Head matching, however, cannot be applied to empty nodes, which are necessary to represent zero anaphora.
A system's predicted counterpart of a gold zero may be absent, spurious, or placed at a different surface position in the sentence, while still expressing the same role.
We therefore employ the \emph{dependency-based method} \citep{oursharedtask2024}, which
 aligns predicted and gold zero mentions within each sentence by maximizing their overlap in the enhanced dependency representation.
Concretely, it solves a one-to-one matching problem in a weighted bipartite graph; each candidate pair is weighted by how well the predicted zero reproduces the gold zero’s dependencies.
Pairs that correctly recover both the parent and the dependency label are rewarded more, while the procedure still works reliably even when labels are missing.

\paragraph{Primary score}
Following common practice in coreference resolution, the primary metric is CoNLL F$_1$  \citep{CoNLL-MELA-score,pradhan-etal-2014-scoring}.
It is defined as the unweighted mean of the $F_1$ scores of three established measures: MUC \citep{MUC-score}, B$^3$ \citep{Bcubed-score}, and CEAF-e \citep{CEAF-score}.
These correspond to complementary viewpoints on coreference quality---link-based, mention-based, and entity-based, respectively.
To identify systems that perform consistently across datasets, we rank submissions by the macro-average of CoNLL F$_1$ over all mini-test sets in the shared task collection.%
%\footnote{%
%The evaluation protocol with macro-averaging CoNLL F$_1$ scores was announced before the start of the development phase and it was used also in previous versions of the shared task.
%We think it is the fairest aggregation method.
%As alternatives, one could average differences to the baseline or average ranks. The former yields the same final ranking as macro-averaging, while the latter would lead to a single difference: in the LLM track, the winner would be LLM-UWB, despite this system not producing output for one dataset and not covering zero anaphora in some datasets (see Sections~\ref{sec:system-submissions} and~\ref{sec:results}).
%}

\paragraph{Supplementary scores}
In addition to the main CoNLL F$_1$ score, we also report variants based on alternative mention matching strategies: partial match%
\footnote{Partial match served as the primary metric in the first edition of the shared task \citep{oursharedtask2022}.}
and exact match.
We further compute a head-match CoNLL score that includes all mentions, including singletons.

For a more detailed view, we provide the individual metrics that make up the CoNLL score (MUC, B$^3$, and CEAF), as well as other popular measures such as BLANC \citep{BLANC-score} and LEA \citep{LEA-score}.
Finally, we report the anaphor-decomposable score for zero anaphora and the mention overlap ratio (MOR), both introduced in \citet{oursharedtask2022}. We also introduce the mention-detection F-score on heads (MD-h).
Both mention-oriented scores assess mention detection quality independently of entity assignment. Whereas MOR assigns each mention pair a graded score based on full-span overlap, MD-h simplifies the comparison to a binary match between mention heads.
