%OLD

% 1) Systematic comparison of non-single-prediction models; reveals that perspective models do not work well.
 
% 2) Further experiments to analyze the reasons: too many labels? no perspective at all.

% To find out why the perspective model did not work, we have to report:
% 1) per worker train/test split imbalance 
% 2) does it work if there are fewer classes (level-1 results)
% 3) Does the perspective model work with clustered annotators?
% Plus how do these compared with existing tasks where perspective models work.

%F: Between the extremes of single-label prediction and the perspectivist approach, IDRR can also be formulated as a multi-label prediction task or a label distribution prediction task. These formulations can be ordered by increasing precision in capturing variability: single (majority) label, multiple labels, label distributions, and finally perspectivism (a separate label for each annotator). It remains unclear which level of precision best reflects implicit DR interpretation.

\subsection{Models}

%In our experiments, we employed both soft label models and specialized architectures for learning from multiple perspectives.

In this section, we provide an overview of all the distinct model types as well and of our baseline.
The perspectivist models employed  were introduced in \citealt{deng2023annotate} (AE) and \citealt{mokhberian-etal-2024-capturing} (AART). These architectures are tailored to predicting personalized labels by means of injecting personalized embeddings into existing generic transformer models.

%\noindent \paragraph{Soft label model:} In our design of a soft label classifier, we took the existing systems for learning from soft labels for IDRR into consideration. These are represented by \citet{costa2024exploring}, \citet{long2024multi}, and \citet{pyatkin-etal-2023-design}, the latter showing promising results on DiscoGEM data. As in these works, we augment a base model with a soft label classification head and learn the distribution of labels directly. Throughout training, the model is optimized using KL-divergence loss.

%\noindent \paragraph{AART:} The AART architecture \cite{mokhberian-etal-2024-capturing} combines text embeddings from pretrained transformer models with learned annotator embeddings. Formally, given a text item $x_i$ and annotator $a_j$, the combined embedding is computed as:

% $$
% g(x_i, a_j) = e(x_i) + f(a_j)
% $$

% where $e(x_i)$ is the text embedding, and $f(a_j)$ is a learned annotator-specific embedding. This combined embedding is then fed into a common classification head. The model employs a multipart loss function consisting of cross-entropy loss, L2 regularization on annotator embeddings, and a contrastive loss designed to cluster similar annotator perspectives.

\noindent \paragraph{Base model:} As all the three architectures we used are model-agnostic and compatible with any encoder transformer, we ensured the consistency of our experiments by using roberta-base \cite{liu2019roberta} uniformly across the runs.

% \noindent \paragraph{Baseline:} To ensure that our perspectivist architectures meaningfully tackle the personalization task, we compare them against a baseline borrowed from \citet{deng2023annotate} labeled as naive concat. This baseline makes use of a non-customized transformer classifier that should yield annotator-aware predictions based on a deliberately sparse signal in the form of text-based annotator identifiers. The intuition behind this is that a specialized architecture can be expected to outperform a generic transformer that only receives concatenated annotator ids throughout training and inference.


\section{Metrics}

Our experiments involved the evaluation of two different types of models: perspectivist models, which may use the same classification metrics as gold label classifiers, and soft label models that require distribution-level metrics. A particularly challenging aspect was benchmarking the models against one another to determine the relative adequacy of either approach. We handled this problem through soft evaluation of perspectivist predictions, whereby annotator-level predictions for a single data point were first aggregated as a distribution and then processed using soft label metrics. At the same time, we also measured the single-label classification metrics, such as F1-score, to ensure that our results are straightforwardly interpretable.
\noindent \paragraph{Soft metrics:}
In the existing literature, the empirical studies employ a variety of metrics for learning from soft labels, as there exists no consensus concerning the optimal metric for that purpose. Due to this, we employ a set of different metrics, which allows us to additionally consider their appropriateness for our task.

Two distribution-level metrics that are widely used in soft label experiments \cite{fornaciari-etal-2021-beyond,rizzi2024soft} are \textbf{Cross-Entropy} and \textbf{Manhattan Distance}. We thus only define these metrics in passing. Cross-Entropy corresponds to minimizing the Kullback–Leibler divergence between the two distributions. Lower values indicate better alignment between predicted and target probability scores. Manhattan Distance, on the other hand,  measures the total absolute deviation between predicted and reference distributions.

A further metric we use is \textbf{Jensen-Shannon Divergence}, which aggregates the two respective KL-divergence scores for the source and the target distributions and thus ensures a symmetric measure. This metric has also made its way into soft label classification both as an evaluation metric and as a loss function \cite{LeWiDi2025}. It is defined as follows:

\[
\mathrm{JSD}(y ,|, \hat{y}) = \tfrac{1}{2} \mathrm{KL}(y ,||, m) + \tfrac{1}{2} \mathrm{KL}(\hat{y} ,||, m).
\]

Finally, the formulations for soft classification scores (\textbf{Fuzzy Precision, Recall, and F1}) introduced in \cite{harju2023evaluating} extend the typical classification metrics to the distribution level based on the insights from the fuzzy set theory. Specifically, for each class $i$, let the reference membership be $y_i$ and the predicted membership be $\hat{y}_i$. Using the minimum operator to define fuzzy intersection:

\[ 
\mathrm{F.Precision} = \frac{\sum_i \min(\hat{y}_i, y_i)}{\sum_i \hat{y}_i} 
\]
\[ 
\mathrm{F.Recall} = \frac{\sum_i \min(\hat{y}_i, y_i)}{\sum_i y_i}, 
\] 
\[ \mathrm{F.F1} = \frac{2 \sum_i \min(\hat{y}_i, y_i)}{\sum_i (\hat{y}_i + y_i)}. \]

We hypothesize that this score can prove to be a viable alternative to distance metrics when evaluating distribution-level models.

\noindent \paragraph{Hard metrics:} We evaluate annotator-level performance using the standard classification metrics, \textbf{Precision}, \textbf{Recall} and \textbf{F1-score}, on a per-annotator basis with post-hoc averaging. The question how well these metrics scale to a multi-perspective setting has been problematized in a number of studies; however, there exists no consensus on what other metric should replace them \cite{frenda2024perspectivist}. Thus, we stick to the well-tested options for the sake of consistency.


\section{Results}

\subsubsection{How well do perspective models do on DiscoGeM?}\label{sec:basic_result}

F: Show results of perspective models WITHOUT POOLING. Use Exact Match accuracy compared with baseline prediction (same prediction to all annotators but evaluated by exact match)

Compare results with other datasets in the Deng paper, showing the annotator and example statistics together

\subsubsection{Is the bad result due to too many classes?}

F: Add results of level-1 classes prediction (to the same table), as if it is another dataset (different annotator and example statistics). Compare with baseline levl model

\subsubsection{Is the bad result due to unbalanced train/test ratios?}

F: Show train/test ratio statistics grouped by class. A table of average train/test ratio and average exact match accuracy per class. (compared with baseline model) Are the bad performing classes due to low train/test accuracy?

Alternatively, a plot of per-annotator performance against train/test ratio per annotator.



\subsubsection{Does it work with higher performing annotators?}

F: Cluster annotators based on agreement with majority labels

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{IDRR_Perspectives/latex/kmean_highagree.png}
    \includegraphics[width=0.8\linewidth]{IDRR_Perspectives/latex/kmean_medagree.png}
    \includegraphics[width=0.8\linewidth]{IDRR_Perspectives/latex/kmean_lowagree.png}
    \caption{Agreement with majority label of 3 groups of annotators (High/medium/low agreement)}
    \label{fig:clusters}
\end{figure}

F: Show exact match results of each group of annotators.
(and results using clustered annotators (just 3 clusters, need to run experiments for this as we only have 9 clusters results) ?)

\subsubsection{Results pooling all annotators and compared with distributional approach}

F: The better results of distributional approach suggest that there are multiple interpretations per item but not consistent perspective.


\begin{table*}
    \centering
    \begin{tabular}{l|llll}
         \hline
         & baseline & multi-label & label-dist & AE \\
         \hline
         Cross Entropy&3.071&2.134&2.253&2.935\\
        JS divergence&0.371&0.342&0.268&0.450\\
        Manhattan dist.&1.126&1.786&0.875&1.118\\
        Euclidean dist.&0.551&0.864&0.343&0.487\\
        %precision&0.332&0.332&0.332&0.333\\
        %recall&1&1&1&1\\
        %F1&0.491&0.491&0.491&0.491\\
        fuzzy prec&0.269&0.408&0.3&0.211\\
        fuzzy recall&0.674&0.594&0.926&0.895\\
        fuzzy F1&0.385&0.484&0.453&0.341\\
        \hline
    \end{tabular}
    \caption{Evaluation of model prediction against DiscoGeM label distributions; results based on 3 random seeds}
    \label{tab:placeholder}
\end{table*}

\begin{table}
    \centering
    \begin{tabular}{l|llll}
        \hline
        & F-1$_{G}$ & F-1$_{A}$ & Acc$_{G}$ & Acc$_{A}$ \\
        \hline
        & & level-1 & & \\
        \hline
        baseline & 0.35 & 0.34 & 0.44 & 0.44 \\
        %multi-label & 0.35 & 0.34 & 0.44 & 0.44 \\
        %label-dist & 0.35 & 0.35 & 0.46 & 0.46 \\
        AE & 0.74 & 0.71 & 0.84 & 0.83 \\
        \hline
        & & level-2 & & \\
        \hline
        baseline & 0.15 & 0.16 & 0.28 & 0.27 \\
        %multi-label & 0.14 & 0.13 & 0.2 & 0.19 \\
        %label-dist & 0.17 & 0.17 & 0.29 & 0.28 \\
        AE & 0.2 & 0.23 & 0.4 & 0.39 \\
        \hline
    \end{tabular}
    \caption{Hard metrics performance.}
    \label{tab:hard}
\end{table}

We report our results, averaged per 3 runs, in Table \ref{tab:hard}. As the comparison shows, the models intended to capture individual annotator bias do not achieve the intended efficiency at predicting personalized labels. 

Partly, this shortcoming can be explained by label imbalance that is exacerbated by learning from perspectives. Specifically, examples tied to rarer connective senses are few overall and are even fewer in each annotator's own batch, as the data was distributed stochastically; moreover, some annotators could receive none of these items at all. Consequently, models mimicking these annotators are unlikely to produce rarer labels even when these are warranted. 
As a result, a faithful approximation of annotator perspectives in DiscoGEM necessitates a strategy to mitigate this issue: e.g., grouping and merging of similar perspectives to increase label coverage or finding an annotator subset that covers


\begin{table}
    \centering
    \begin{tabular}{l|llll}
        \hline
        & F-1$_{G}$ & F-1$_{A}$ & Acc$_{G}$ & Acc$_{A}$ \\
        \hline
        & & level-1 & & \\
        \hline
        baseline & 0.337 & 0.325 & 0.428 & 0.428 \\
        multi-label & 0.342 & 0.324 & 0.432 & 0.43 \\
        label-dist & 0.346 & 0.323 & 0.452 & 0.433 \\
        AE$_{kmeans}$ & 0.571 & 0.524 & 0.685 & 0.675 \\
        AE$_{agglom.}$ & 0.604 & 0.584 & 0.728 & 0.726 \\
        \hline
        & & level-2 & & \\
        \hline
        baseline & 0.151 & 0.145 & 0.274 & 0.261 \\
        multi-label & 0.138 & 0.131 & 0.2 & 0.191 \\
        label-dist & 0.158 & 0.149 & 0.283 & 0.273 \\
        AE$_{kmeans}$ & 0.144 & 0.123 & 0.331 & 0.327 \\
        AE$_{agglom.}$ & 0.151 & 0.141 & 0.348 & 0.354 \\
        \hline
    \end{tabular}
    \caption{Cluster performance, N=9}
    \label{tab:clustering9}
\end{table}

\begin{table}
    \centering
    \begin{tabular}{l|llll}
        \hline
        & F-1$_{G}$ & F-1$_{A}$ & Acc$_{G}$ & Acc$_{A}$ \\
        \hline
        & & level-1 & & \\
        \hline
        baseline & 0.326 & 0.301 & 0.409 & 0.388 \\
        multi-label & 0.319 & 0.295 & 0.407 & 0.387 \\
        label-dist & 0.33 & 0.303 & 0.433 & 0.41 \\
        AE$_{kmeans}$ & 0.334 & 0.33 & 0.488 & 0.495 \\
        \hline
        & & level-2 & & \\
        \hline
        baseline & 0.135 & 0.12 & 0.253 & 0.224 \\
        multi-label & 0.12 & 0.105 & 0.178 & 0.155 \\
        label-dist & 0.145 & 0.128 & 0.263 & 0.232 \\
        AE$_{kmeans}$ & 0.086 & 0.073 & 0.216 & 0.195 \\
        \hline
    \end{tabular}
    \caption{Cluster performance, N=3}
    \label{tab:clustering3}
\end{table}


\subsubsection{Qualitative analysis}

F: Analyze a subset of items labeled by the same few annotators to show, if possible, that the perspectives are not consistent.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



% Copied from another draft
DG 1.5 label distribution:

\begin{table}[ht]
\small \centering
  \begin{tabular}{@{}l@{}|ll@{}|lll@{}|l@{}}
         \hline
         & \multicolumn{2}{c|}{PDTB 3.0}     
         & \multicolumn{4}{c}{DG 1.5 (test)} \\
         \cline{2-7}
         & train & dev & EP & WK & NV & ttl.\\       
        
         
         \hline\hline
         \textbf{Expansion}&&&&&\\
\textsc{conjunc.}&3584&298&314&177&323&814\\
\textsc{lev.-of-det.}&2493&262&532&161&460&1153\\
\textsc{instant.}&1117&116&212&37&94&343\\
\textsc{manner}&191&14&20&0&6&26\\
\textsc{substitut.}&278&27&41&4&47&92\\
\textsc{equival.}&252&25&50&2&38&90\\
\textsc{disjunct.}&0&0&0&0&5&5\\
\textsc{exception}&0&0&0&1&6&10\\
%\textsc{disjunction}&22&5&0&0&5&5\\
%\textsc{exception}&4&1&3&1&6&10\\

        \hline \textbf{Contingency}&&&&&\\
\textsc{cause}&4469&450&885&86&857&1828\\
\textsc{purpose}&1102&97&139&10&49&198\\
\textsc{cause+bel.}&157&13&0&0&0&0\\
\textsc{condition}&152&18&85&3&19&107\\


        \hline \textbf{Contrast} &&&&&\\
\textsc{concession}&1164&103&229&23&186&438\\
\textsc{contrast}&639&82&38&19&58&115\\
\textsc{similarity}&0&0&65&6&42&113\\
%\textsc{similarity}&21&4&65&6&42&113\\

        \hline \textbf{Temporal}&&&&&\\
\textsc{asynchr.}&985&102&18&70&536&624\\
\textsc{synchr.}&433&34&73&16&158&247\\
%\hline
%\textsc{no relation}&0&0&80&30&176&286\\
\hline\hline
\textbf{Total}&17016 & 1641 &2704 &615&2884&6203 \\
%\textbf{Total}&17016 & 1641 & 2784 & 645 & 3060 & 6489 \\
           \hline
    \end{tabular}

   \caption{Distribution of the Level-2 classes, grouped under 4 Level-1 categories, in each data subset. DiscoGeM's distribution is based on the single majority label per sample
   \label{tab:dist}}
    \end{table}