\section{Introduction}

% dataset cartography is largely influenced by label frequency, this effect is exacerbated in sparser settings
% minority labels predominantly display low variability and confidence
% level 3 performs better, since the easy subset is easier to disentangle
The task of discourse annotation remains a 
%long-standing 
challenge in data collection methodology. Scalable annotation solutions (crowdsourcing; silver data) cannot fully ensure the desired high data quality; on the other hand, expert annotation lacks scalability due to high costs \cite{das-etal-2017-good,Scholman2017CrowdsourcingDI,yung2019crowdsourcing}. 

With crowdsourced discourse annotation, maintaining a balance between a descriptive and a prescriptive approach is particularly challenging. 
%When using crowdsourced discourse labels, the choice between the descriptive and the prescriptive paradigm introduces a further challenge.
Although there is a need for error filtering, 
a prescriptive approach is hard to justify, as the annotation schemes tend to treat diverging interpretations permissively \cite{prasad2017penn}, while errors are hard to tease apart from plausible readings:

\newcounter{excounter}
\newcommand{\ex}[1]{%
  \refstepcounter{excounter}%
  \begin{list}{}{\leftmargin=1.5em \rightmargin=0pt}%
  \item[(\theexcounter)] {\small #1}%
  \end{list}%
}
\ex {Arg1: \textit{This time he did not go back to the boat.} %\\
Arg2: He sat down in the dark by Bilbo. %\\
A)~\textsc{substitution} / B)~\textsc{cause}}

Dataset cartography \citep{swayamdipta_dataset_2020} 
would appear to be a useful analytic tool 
in these circumstances, 
%for this type of annotation, 
%could offer a solution, 
as it has 
proven effective 
%shown particular efficiency 
at detecting noisy labels in crowdsourced datasets \cite{klie_annotation_2023}. Dataset cartography hypothesizes 
%Specifically, the proposed approach suggests 
that data instances that appear \textit{hard-to-learn} from their training dynamics often coincide with annotation errors (see \citealp{anand_dont_2024} for a counter-argument). As such, they could be removed to improve model convergence or avoided in active learning \cite{zhang2021cartography}. Likewise, \textit{easy-to-learn} data should not be prioritized in training.

\begin{figure*}
    \centering
     \includegraphics[width=0.52\linewidth]{IDRR_Perspectives/figures/lvl3.5_AE_label_breakdown.jpg}
     \includegraphics[width=0.47\linewidth]{IDRR_Perspectives/figures/lvl3.5_AE_conf_vs_var_color_agreement_factor.jpg}
    \caption{Data maps for the Annotator Embeddings model \cite{deng-etal-2023-annotate} under high sense granularity (level-3). X- and Y-axis correspond to \textbf{variability} and \textbf{confidence}, respectively. Plot colors, left: labels; right: agreement (ratio of the majority label). We note that high-agreement area is dominated by \textit{precedence} (gray color).}
    \label{fig:AE_agr_small}
\end{figure*}


For our case study, we use DiscoGeM \cite{scholman-etal-2022-discogem,yung-demberg-2025-crowdsourcing},
a %the first 
large corpus annotated in the Penn Discourse Treebank standard \cite[PDTB,][]{AB2/SUU9CB_2019}. Notably, unlike PDTB or most other discourse resources, DiscoGeM retains a wide spectrum of crowd-collected labels for each item: 10+ labels. Thus, it offers a realistic image of label distribution in the PDTB standard; similarly, it offers insights into crowd workers' individual labeling behavior. At the same time, DiscoGeM incorporates noisy labels in some proportion, which stresses the need for automatic assessment of data quality \cite{yung_human_2026}.

DiscoGeM challenges dataset cartography in several ways. 
First, prior studies have shown that diverging annotations in DiscoGeM can at least partly be attributed to individual, subjective biases \cite{pyatkin-etal-2023-design}. 
\citet{anand_dont_2024} show that items with subjective variation, while not erroneous, can still fall into the \textit{hard-to-learn} region of single-label classifiers; 
in response, %in response to this, 
we 
use %map the dataset using 
both single-label and annotator-aware classifiers \cite{Cabitza_Campagner_Basile_2023}. 

A second challenge is posed by label imbalance, as infrequent labels often constitute a small fraction of the DiscoGeM data \cite{yung-demberg-2025-crowdsourcing}.
E.g., at level-3, the most granular level of the PDTB sense hierarchy, 9 out of 29 classes are picked by the majority in less than 1\% of cases. It can thus be expected that infrequent classes will display low confidence, which indeed occurs in practice. Due to subjectivity and label imbalance, \textit{hard-to-learn} IDRR data cannot be pruned indiscriminately, as it combines genuinely erroneous, subjective and merely infrequent labels.

At the same time, our experiments also help detect a persistent \textit{easy-to-learn} subset that strongly affects model training and evaluation -- namely, instances of the \textit{precedence} relation supported by strong lexical or syntactic cues; e.g., a temporal dependent clause at the start of Arg2:

\vspace{-5pt}
\ex {Arg1: \textit{...While it had devastated me and stopped me in my tracks, for Maxime, it had broken down barriers, released him from a straitjacket, and left him free to write his own story.}
Arg2: After what happened, I was never the same...}
\vspace{-5pt}

This set is globally distinguishable, but more so when isolated: e.g., when training at levels 2 and 3 using the annotator-aware architectures; this accounts for annotators not following the cues. 
% This allows us to explain another notable trend: namely, that models that we trained and evaluated on level-3 labels (29 classes) perform on par with the models we trained on level-2 (17 classes), which conflicts with the common intuition that coarser labels are easier to learn \cite{xiang2023survey}.

Overall, we make the first attempt at studying crowdsourced IDRR data through the lens of training dynamics. We demonstrate that filtering the \textit{hard-to-learn} data in DiscoGeM proves too difficult, but \textit{easy-to-learn} instances expose heuristics that should be handled explicitly in data collection and in model training. 
Finally, we observe that perspectives and class granularity both determine the sharpness of class boundaries, which effectively enables improved confidence on granular labels.

% Disagreement-aware approaches in data collection and machine learning have earned recognition in various NLP tasks \cite{basile2020s,,rottger-etal-2022-two,},
% %One such methodology is data perspectivism  \cite{basile2020s,Cabitza_Campagner_Basile_2023}, which advocates for a descriptive approach to data collection \cite{rottger-etal-2022-two}, 
% particularly in tasks involving subjectivity and linguistic ambiguity. 
% Specialized architectures have been proposed to model individual annotator stances instead of the ground truth in such domains as hate speech detection and toxicity detection \cite{davani-etal-2022-dealing,kanclerz-etal-2022-ground,ngo-etal-2022-studemo,deng-etal-2023-annotate,mokhberian-etal-2024-capturing}. 
% However, this approach has potentially problematic implications, as human-produced data is known to be prone to errors or deliberate attempts at brute-forcing the task by annotators \cite{poesio&artstein:ACL-ANNO-05,plank2014linguistically,plank-2022-problem}. The applicability of perspectivism is therefore task-dependent. 
% %many traditional NLP tasks, including shallow discourse parsing and IDRR.
% In particular, since variation in DR interpretation often stems from individual differences in cognitive ability \cite{sanders-etal-2022-towards, SpoorenDegand+2010+241+266}, the applicability of existing perspectivist models tailored for subjective tasks remains an open question.

