\documentclass[11pt]{article}

% Table formatting 
\usepackage{booktabs}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}

% comments enviornment 
\usepackage[dvipsnames]{xcolor}

\newenvironment{comment}
  {\color{WildStrawberry}\bfseries [COMMENT] \normalfont\color{WildStrawberry}}
  {}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\usepackage{amsmath}

\title{ZYC at SemEval-2026 Task 5: Application of BERT-based Contextual Embeddings Similarity for WSD}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Sunny Zhou \\
  \texttt{sunny.t.zhou@gmail.com} \\\And
  Jordan Youner\\
  \texttt{jordan.youner@gmail.com} \\\And
  Dean Cahill \\
  \texttt{dpcahill98@gmail.com} 
  }

%\author{
%  \textbf{First Author\textsuperscript{1}},
%  \textbf{Second Author\textsuperscript{1,2}},
%  \textbf{Third T. Author\textsuperscript{1}},
%  \textbf{Fourth Author\textsuperscript{1}},
%\\
%  \textbf{Fifth Author\textsuperscript{1,2}},
%  \textbf{Sixth Author\textsuperscript{1}},
%  \textbf{Seventh Author\textsuperscript{1}},
%  \textbf{Eighth Author \textsuperscript{1,2,3,4}},
%\\
%  \textbf{Ninth Author\textsuperscript{1}},
%  \textbf{Tenth Author\textsuperscript{1}},
%  \textbf{Eleventh E. Author\textsuperscript{1,2,3,4,5}},
%  \textbf{Twelfth Author\textsuperscript{1}},
%\\
%  \textbf{Thirteenth Author\textsuperscript{3}},
%  \textbf{Fourteenth F. Author\textsuperscript{2,4}},
%  \textbf{Fifteenth Author\textsuperscript{1}},
%  \textbf{Sixteenth Author\textsuperscript{1}},
%\\
%  \textbf{Seventeenth S. Author\textsuperscript{4,5}},
%  \textbf{Eighteenth Author\textsuperscript{3,4}},
%  \textbf{Nineteenth N. Author\textsuperscript{2,5}},
%  \textbf{Twentieth Author\textsuperscript{1}}
%\\
%\\
%  \textsuperscript{1}Affiliation 1,
%  \textsuperscript{2}Affiliation 2,
%  \textsuperscript{3}Affiliation 3,
%  \textsuperscript{4}Affiliation 4,
%  \textsuperscript{5}Affiliation 5
%\\
%  \small{
%    \textbf{Correspondence:} \href{mailto:email@domain}{email@domain}
%  }
%}

\begin{document}
\maketitle



\begin{abstract}
We investigate contextual embedding manipulation for Word Sense Disambiguation (WSD) as part of SemEval-2026 Task 5. We propose four approaches built on BERT-like pretrained models, experimenting with the informativeness of similarity calculations and classification methods. We introduce scratch-trained cross-attention mechanisms inspired by GLiNER to compute similarity between definition or synonym representations and the full context. Our best performance achieved 57\% accuracy with a Spearman correlation of 0.20. Our results suggest that finetuning strategy and training curriculum matter more than pretrained model choice for this novel task, and we identify several directions for future improvement. View our code base at: \url{https://github.com/heliosraz/SemEval52026}
\end{abstract}
% System description papers should focus on:

% Replicability: present all details that will allow someone else to replicate your system
% Analysis: focus more on results and analysis and less on discussing rankings; report results on several runs of the system (even beyond the official submissions); present ablation experiments showing usefulness of different features and techniques; show comparisons with baselines.
% Duplication: cite the task description paper; you can avoid repeating details of the task and data, however, briefly outlining the task and relevant aspects of the data is a good idea. (The official BibTeX citations for papers will not be released until the camera-ready submission period, so during the initial submission, please use some sort of placeholder citation.)

\section{Introduction}
Polysemy is an extremely common phenomenon in natural language, and creates a great deal of complexity for accurately modeling language. In this paper, as a part of SemEval 2026 Task 5 \cite{semeval2026-task-5}, we attempt to tackle this topic. We investigate the efficacy of contextual embedding manipulation strategies on this task by using four approaches, based on BERT and SBERT-like models. (1) Our naive approach is a scales plausibility score between contextual embeddings of the target word in both an ambiguous context and an unambiguous example sentence, based on cosine similarity. (2) We train a classification head to predict the score from the similarity between the ambiguous context and definition using SBERT.

Our remaining two approaches introduce cross-attention as a mechanism for calculating similarity. In both, we adopt \citealt{zaratiana-etal-2024-gliner}'s method of training a similarity head from embeddings generated from sequence pairs. (3) in Definition Cross-Attention (DXA), this is done with the ambiguous word's definition, and the full instance context. (4) in Synonym Entity Recognition (SER), we represent the senses of a target word indirectly as categories with WordNet \cite{miller-1994-wordnet} synonyms and compare them to the context.

Our final submission reflects the performance of our four approaches tested with a variety of pretrained models. Each approach builds on the last under the assumption of capturing contextual knowledge. Our approaches achieve a max accuracy of 57\% with a Spearman's correlation of 0.20 on the test set. The results suggest limitations within our method, and we propose a variety of improvements for the future.


\section{Related Work}

\subsection{Embedding Similarity for WSD}
WSD is commonly tackled as a constituent problem to be addressed in other, more comprehensive tasks, such as Neural Machine Translation. Sense similarity has been a common approach for tackling WSD for decades \cite{lesk1986}. The rise of embedding spaces as a way to reason about relative meaning has provided for fruitful exploration into this notion of similarity, and others have produced various forms of embeddings designed to capture this semantic information. \citealt{suzuki2018} developed concept embeddings for all-words WSD, fully disambiguating entire documents by p
redicting similarity between target words and their surrounding contexts across different senses.

Gloss knowledge has also been shown to influence model performance on WSD \cite{luo2018}. When it comes to using models like BERT for word sense disambiguation, we must choose how we leverage both contextual and gloss information (i.e correct word sense). Given a context and a set of possible meanings, one might expect that the proper meaning has a higher similarity. In other words, we can treat WSD as a classification task on context-sense pairs based on embedding similarity. \citealt{huang-etal-2019-glossbert} utilize BERT in such a manner. They build a set of context-possible gloss pairs with the Semeval 2007 WSD dataset and the senses present in WordNet, labeling pairs with yes or no, and they achieve high performance on contemporary benchmarks. Our approaches are informed by this notion of leveraging the similarity between these embeddings. In particular, one of our approaches involves \citealt{reimers2019sentencebertsentenceembeddingsusing}'s Sentence Transformer (SBERT) model to experiment with pooling the sense embeddings as another way to capture this gloss knowledge.

\subsection{Model Training Practices}
It is well known that BERT and variations utilize certain pretraining objectives, such as \textbf{Masked Language Modeling (MLM)}, \textbf{Next Sentence Prediction (NSP)}, and Permuted Lanugage Modeling (PLM). These methods are shown to enable BERT to generalize over task specific training later \cite{liu2019robertarobustlyoptimizedbert, he2021debertadecodingenhancedbertdisentangled, song2020mpnetmaskedpermutedpretraining, devlin-etal-2019-bert}.

\citealt{howard-ruder-2018-universal} develop a system for Universal Language Model Fine-Tuning (ULMFiT), which enables generalized transfer learning across various text classification tasks. A language model is pretrained on general-domain data, finetuned on task-specific data with the same pretraining task, and then a general classifier block is added. The finetuned classification uses a gradual unfreezing of each layer, starting from the final one and unfreezing the preceding layer with each epoch. This approach prevents catastrophic
forgetting, which is common in small dataset finetuning.

Our approach in DXA and SER is inspired by this staged training. In both strategies, we separate the training of the attention head and the classifier block, and we utilize gradual unfreezing when training the classifier. DXA, in particular, implements the MLM pretraining from BERT and classifier finetuning from ULMFiT.
% While we are training our attention head from scratch, we do so on task-specific data. 


% [TODO]:
%     - Write about ULMFiT and GLiNER
%     - Maybe the masked LM pretrinaing belongs here
%     - describe ULMFit/GLINER approach
%     - note what we take from this?
%     -GLiNER- NER approach using bidirectional transformers, generalizing NER task to associate entity types with text spans in the same embedding space.
%     -They use a pretrained text encoder (bidirectional LM such as BERT, specifically seems like they used DeBERTa ), feed input word embeddings (sentence, not list of ent types) into a span representation layer, and feed entity type embeddings into ffn. then compute matching score between entity and span representations using dot product + sigmoid.
%     - The actual input to the model includes an arbitrary list of entity types separated by a learned [ENT] tag, followed by the actual input text.
%     - their model achieves higher average performance (across categories) of out of domain NER than its contemporaries, and also higher average performance across 20 NER datasets. 

%    - ULMFiT-proposed universal fine tuning architecture to enable generalized transfer learning across NLP. first a language model is trained on general domain data, then it is finetuned using "discriminative finetuning" (where each layer can be tuned with different learning rates) and "slanted triangular learning rates" (where LR linearly increases to a point and then linearly decreases). Finally, two linear blocks are added for the classifier, and these values are trained from scratch (as the "classifier finetuning" portion), utilizing a "gradual unfreezing" strategy wherein each layer is unfrozen starting from the last, and trained for one epoch.
%    - their work reduces error rates across the board for six datasets in three tasks- sentiment analysis, question classification, and topic classification. error rate reduced by 18-24% on most datasets.
%    - Our architecture is inspired by both GLiNER and ULMFiT. Leveraging the 
%    - Like ULMFiT we separated the classifier block from the pretrained model and trained in stages
%    - instead of directly fine tuning LM we created attention head from scratch and trained it on the task-specific data
%    - we fed the pretrained model embeddings into the attention head
%    - then we trained our classifier with gradual unfreezing


\section{Task Description}

This task asks us to produce a plausibility score [1, 5] for a semantic sense that matches human annotators' judgments. A precontext and an ambiguous sentence (and sometimes an ending) is given for each instance. It is presented as a discrete choice to the annotators, and thus we choose to approach this problem as a classification task wherein the predicted class is a given score; our goal is to match the prediction distribution to the annotators' distribution.

\section{Data and Evaluation Metric}

The task organizers provided AmbiStory, a pre-split dataset containing 2279 raw instance in train, 587 in dev, and 929 in test \cite{semeval2026-task-5}. To provide the word in the ambiguous sentence enough information to work with, we combined the pre-context, ambiguous sentence, and endings to form the complete story, which we call full context. Additionally, an average and a standard deviation of the human judgments were given for each instance. We use this to calculate the annotator's unnormalized normal distribution across a 5-point plausibility score.

We further split the train data by crossing "judged\_meaning" with the set "full\_context", "ending", and "example\_sentence" and the set "full\_context", "ending" with itself. Each of these pairs were labeled as "target" and "source" respectively. This yielded a more general dataset of 12159 instances used for pretraining in DXA. 
The full context and ending self crossed pairs and the judged meaning and example sentence pairs were given an average of 5 and a standard deviation of 0. Since the models produce 0-indexed classifications, all of the averages are decreased by 1.

The task calculates correctness based on if a prediction is within a standard deviation of the average. To match this continuous metric, we calculate the average from the classification distribution to use as our prediction.

\section{Methodology}

\subsection{Naive Similarity}
We input the full context into BERT, which updates the embeddings with the ambiguous contextual information. The given example sentence is assumed to be an unambiguous use of the target homonym, capturing the same sense to the judged meaning, so cosine similarity measures the difference in sense usage. %so we take the cosine similarity between its embedding and that of the context. 
%clarificaiton required i think^
Similar to how functions (e.g. softmax) offer probability proxies, we use this calculation to proxy model's probability confidence that these two embeddings match meanings. Since we are comparing two very similar senses, we can assume that  the cosine similarity is bounded by $[0, 1]$, instead of $[-1,1]$.  Scaling this confidence with $5$ and rounding up to the nearest integer, we map this to a plausibility score of $[0, 5]$.

\subsection{Cross Context Embeddings (CCE)}
The previous approach makes a critical assumption that sense captured by the example sentence proxies the judged meaning's sense. However, an ambiguous word may still retain ambiguity even in the clearest of contexts. Thus, we posit that a weighted pooling of the embeddings in judged meaning delivers a more accurate representation of the target meaning.

Additionally, while the previous approach uses embedding similarity to map directly to human plausibility judgment, we ask here if there exists a projection from embedding similarity to a plausibility score. Thus, this approach consists of four modules: sentence transformer, context transformer, similarity, and classifier.

\subsubsection{Sentence Transformer Module}
We calculate and pool the contextual embeddings of the judged meaning with a pretrained SBERT-like model in this module. For an embedding dimension of E, the resultant vector is shape $1\times E$.
\subsubsection{Context Transformer Module}
We gather contextual embeddings for the full context, making this module rather similar to the naive similarity approach. However, unlike the naive similarity approach, the model used here needs to match the SBERT-like model used in the previous module, but without pooling, ensuring the embedding sizes are the same. The resultant matrix shape is $S\times E$, where $S$ is the sequence length and $E$ is the embedding dimension.
\subsubsection{Similarity Module}
We batch multiply the context transformer module result and the sentence transformer module result to calculate the similarity, preserving angle and magnitude, as opposed to only angle in cosine similarity \cite{vaswani2017}. The resultant vector is shape $S\times 1$.
\subsubsection{Classifier Module}
The $S\times 1$ vector is fed into a general classifier with one hidden layer size $128$ and a ReLU activation and dropout layer after the hidden layer. The classifier outputs a distribution over five classes, representing the probability of each plausibility score.


\subsection{Definition Cross-Attention (DXA)}
The previous approach relies on SBERT pooling to capture the meanings in judged meaning, limiting the contentfulness of the signal. \citealt{zaratiana-etal-2024-gliner} addresses this disconnect between the pretrained model and trainable signals by mapping contextual embeddings from a BERT-like model to a space where similarity is meaningful. Instead of searching for entity spans, a word sense disambiguation task can be viewed as searching for important parts of a definition within a context, utilizing the definition as a expanded proxy of the target sense.

We implement this using cross-attention, where keys are from the context, and queries and values are from the words within the definitions. This approach consists of three modules: encoder, cross-attention, and classifier module. The cross-attention module replaces the similarity module from CCE, and the classifier module is identical to CCE's.

\subsubsection{Encoder Module}
A BERT-like module is used to gather the contextual embeddings of an input with the following construction, borrowed from \citealt{zaratiana-etal-2024-gliner}:

\begin{small} \begin{verbatim}
    [JUDGED_MEANTING]+[SEP]+[FULL_CONTEXT]
\end{verbatim} \end{small} 
Judged meaning and full context are both padded separately to a length of 255 and 256 respectively, yielding a sequence length of 512 regardless of judged meaning or full context length.

\subsubsection{Cross-Attention Module}
Following the approach in \citealt{zaratiana-etal-2024-gliner}, the encoded input is split at the $[SEP]$ token, resulting in two sequences: $x_{target}$ and $x_{source}$. Cross-attention is then calculated, following PyTorch's \verb|scaled_dot_product_attention()| implementation:
\begin{align*}
     q &= W_q@x_{target}\\
     k &= W_k@x_{source}\\
     v &= W_v@x_{target}\\
     H &= softmax(\frac{q@k^T}{\sqrt{d}})@v
\end{align*}
where $d$ is the size of $k$, and $H$ the output of the attention.
The logits are compiled by concatenating the max and mean across the keys \cite{howard-ruder-2018-universal}:
$$h = [max(H), mean(H)]$$
The resultant vector is shape $1\times 2*L_{source}$

\subsection{Synonym Entity Recognition (SER)}
\citealt{zaratiana-etal-2024-gliner}'s addition of the trainable [ENT] tag to facilitate the NER task in a BERT-like system functions as a sort of latent carrier for the entity type, allowing for a sigmoid to be calculated between the relevant spans and for entities to be properly captured \& scored. We explore, in SER, the potential of quasi-clustering to measure similarity of context across multiple word senses. The intuition of this is that the plausibility of a given sense would correlate with the similarity between a context and a set of senses as a whole. Plausibility score will then, in theory, correlate with the variance of similarity scores across the sense represented by the synonyms.

The general structure of this model's architecture is largely similar to DXA. The following elucidates changes between DXA and SER.

\subsubsection{Encoder Module}
The encoder module primarily differs through the inclusion of a synonym generation step. Pulled from WordNet, the set of senses are concatenated into a vector consisting of alternating words and $[SYN]$ tags. An additional trainable token $[MISC]$ is added to the end of this sequence to capture other senses not covered by the selected synonyms, resulting the from:

\begin{small} \begin{verbatim}
    [[SYN], w_1, ... [SYN], w_n, [MISC]]
\end{verbatim} \end{small} 

These values were tokenized and concatenated to the relevant data from the training instance and then padded to length of 512. The final form of the contextual embeddings of an input are as follows:

\begin{small} \begin{verbatim}
    [SYNONYMS]+[SEP]+[FULL_CONTEXT]+[JUDGED_MEANING]
\end{verbatim} \end{small} 


\subsubsection{Cross-Attention Module}
We select only $[SYN]$ and $[MISC]$ embeddings in the encoded target sequence. The judged meaning portion of the source sequence is mean pooled and joined at the end of the encoded full context. Thus the two sequences are represented as such: 
\begin{align*}
    x_{target} &\rightarrow [[SYN],[SYN],...[MISC]]\\
    x_{source} &\rightarrow [FULL\_CONTEXT]\\
    &+mean([JUDGED\_MEANING])
\end{align*}
The cross-attention mechanism in this module is identical to its counterpart in DXA. The synonym entity index with the max similarity with the judged meaning is used to select the attention weight row used in classification:
$$
    h = [H[argmax(H[:, -1]), :L_{FULL\_CONTEXT}]]
$$
where $H$ is the result from attention and $L_{FULL\_CONTEXT}$ is the length of the full context.
This setup facilitates the computation of a cross-attention between the positions representing each synonym and the full context of the ambiguous sentence.


\section{Experimental Setup}
\subsection{General Setup}
The attention and classifier weights were initialized with Xavier. We trained all of the models for 100 epochs (or until convergence) with AdamW ($\beta_1=0.7$, $\beta_2 = 0.99$) and used KL Divergence for training loss to match the predicted class distribution to the provided distribution. We used a learning rate of $1e^{-5}$ for the encoder layers and $3e^{-5}$ for others. A weight decay of $0.1$ on the encoder layers and $0.01$ on others was also used. These hyperparameters were chosen to reflect \citealt{howard-ruder-2018-universal} and \cite{zaratiana-etal-2024-gliner}'s paradigm. Evaluation uses the same hyperparameters, expect for dropout.

\subsection{DXA}
The training was split into 2 parts, as opposed to the 3 parts in \citealt{howard-ruder-2018-universal}: (1) pretraining of the encoder and attention mechanism, and (2) finetuning of the classifier. Preliminary testing indicated that straight classifier finetuning without pretraining the attention mechanism prevented the capture of any generalizable information, resulting in overfitting

Similar to BERT, masked language modeling (MLM) was used to pretrain. For each instance, a random token was chosen in the definition was masked at index i. Then a possible token is predicted from the ith row of cross-attention module, representing the values of the masked token. $0.4$ dropout was applied to the attention mechanism and $0.3$ dropout was applied to the classifier.

In the classifier finetuning phase, we used gradual thawing to ensure training started from the least generalized components first \cite{howard-ruder-2018-universal}.

\subsection{SER}
This approach utilized a similar methodology as DXA, but without pretraining the cross-attention. We figured the problem with the attention weights here cannot be addressed by a generalizing task, like MLM, but this is not within the scope of this paper. We use the same hyperparameters as in the DXA gradual thawing here.

\section{Results}
While Spearman correlation is not significant, our models overall outperform the naive similarity approach by up to 6\% in terms of accuracy, as seen in Table \ref{tab:results}. The models are ranked from CCE, DXA, to SER, from best to worst. The CCE models generally seem to perform the best, regardless of pretrained model choice. Out of those models, CCE-MPNet was the highest performer with 57\% accuracy and a Spearman's correlation of 0.20.

\begin{table}[h!]
\resizebox{\columnwidth}{!}{

    \begin{tabular}{l l l l}
    \toprule
    \textbf{Model} & \textbf{Accuracy} & \textbf{Spearman} & \textbf{Average}\\
    \midrule 
    \multicolumn{4}{l}{\textbf{Baseline}} \\ 
    GPT-4o* & & & 0.756\\
    Llama-3.1 8B* & & & 0.563\\
    Naive Similarity & 0.51 & 0.21 & 0.36\\
    \midrule
    \multicolumn{4}{l}{\textbf{CCE}} \\ 
    DistilRoBERTa & 0.52 & -0.01 & 0.26\\ 
    MiniLM & 0.57 & 0.04 & 0.31\\
    MPNet & \textbf{0.57} & \textbf{0.20} & \textbf{0.385}\\
    RoBERTa & 0.57 & 0.10 & 0.335\\
    \midrule 
    \multicolumn{4}{l}{\textbf{DXA}} \\
    BERT & 0.53 & 0.07 & 0.30\\
    DeBERTa & 0.53 & -0.02 & 0.26\\
    DistilBERT & 0.52 & 0.04 & 0.28\\
    \midrule 
    \multicolumn{4}{l}{\textbf{SER}} \\ 
    BERT & 0.50 & -0.33 & 0.09\\ 
    DeBERTa & 0.52 & 0.03 & 0.28\\ 
    DistilBERT & 0.52 & 0.15 & 0.34\\
    \bottomrule
    \end{tabular}
    }
    \caption{Task Performance Results. \\ \footnotesize{* Baselines provided by the task organizers.}}
    
    \label{tab:results}

    
\end{table}
 

\section{Discussion}

\subsection{Similarity Module Attention Weights}
By tagging the tokens by part of speech (POS) and looking at the scores produced by each model's similarity module,\footnote{Other attention weights can be found on the GitHub repo.} we can take a closer look at how the models process token relationships. Additionally, by grouping up instances into positive, neutral, and negative samples to remove inverse relationships between the groups, we expect to have low dispersion. 

The models' similarity modules highlight some shortcomings of the models. As expected, we do see lower scores towards negative samples and higher scores towards positive samples, but, surprisingly, this does not necessarily translate to better performance. For example, in Figures \ref{fig:dxa_mean_negative} and \ref{fig:dxa_mean_positive}, we see that DXA-BERT had similarly low scores in both positive and negative samples. In comparison, DXA-DistilBERT has a significantly more contentful attention weights for positive samples, such as its proper noun (NNP) and noun (NN) rows. Despite this, DXA-BERT performs better overall. This indicates that the attention mechanism is not capturing the desired query-key relations. We suspect high scores in negative samples indicate knowledge, such as a lack of similarity, that will help with determining a non-example.

\begin{figure}[h!]
    \centering
    \includegraphics[width=1\linewidth]{figures/dxa_mean_negative.pdf}
    \caption{Mean of [full\_context, judged\_meaning] \textbf{Negative} Sample Pairs Similarity Scores for DXA}
    \label{fig:dxa_mean_negative}
\end{figure}
\begin{figure}[h!]
    \centering
    \includegraphics[width=1\linewidth]{figures/dxa_mean_positive.pdf}
    \caption{Mean of [full\_context, judged\_meaning] \textbf{Positive} Sample Pairs Similarity Scores for DXA}
    \label{fig:dxa_mean_positive}
\end{figure}

On the other hand, the dispersion, measured by the coefficient of variance (CV), illustrates the models' consistency for each sampling group. We found that an inverse relationship between CV and a model's Spearman. In Figure \ref{fig:dxa_cv_negative}, we see that CCE-DistilRoBERTa has by far the most CV and has a negative Spearman. This pattern is observed in nearly all of the sampling groups for all the approaches, suggesting a method of reducing variableness of this mechanic would also increase Spearman across all approaches.
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\linewidth,]{figures/cce_cv_positive.pdf}
    \caption{CV of [full\_context, judged\_meaning] Positive Example Pairs Similarity Scores for CCE}
    \label{fig:dxa_cv_negative}
\end{figure}

Surprisingly, there was not a strong pattern between the feature pairs we compared. POS, such as cardinal numbers and wh-words, were attended to regardless of model performance. This suggests that methods of reducing this noise could improve the classification signal. The better overall performance of the CCE models, despite only having 1D similarity calculation, indicates that pooling with value and the choice of classification signal in DXA and SER dilutes the informativeness of the attention weights, over inflating the noise.

\subsection{Training Curriculum}

Notably, the models perform similarly. The consistency within architecture suggests that the differences among pretrained models do not significantly affect the results. This indicates that model performance on this novel task is less dependent on the pretraining regimen and more on the finetuning. This is surprising. Previous work suggests promising knowledge transfer capabilities for many tasks, implying a resistance to forgetting. 


Moreover, the lacking performance from DXA and SER compared to CCE highlights the difference of the models' initial naivete. In other words, DXA and SER required training of an attention layer from scratch. As mentioned earlier, gross overfitting (e.g. train: 0.98 and val: 0.4 accuracy) in initial training experiments on DXA indicated the training paradigm sensitivity these models have. On the other hand, we reason the more informative similarity calculation in DXA and SER causes the lower prediction variance between base model. This suggests that with a better training curriculum DXA and SER could out perform CCE.

The clearest next steps are a further exploration of the training curriculum and a more robust approach to constructing the "meaningful" embedding space. Although MLM has been fruitful for previous BERT-like models, the success of GLiNER suggests explicit training of the keys, queries, and values could be useful instead of relying on a classification head to translate a meaningful signal. A better training curriculum could also consolidate DXA and SER as SER requires the understanding of judged meaning in addition to the synonym senses. Implementing pretraining in SER, in order to capture other forms of information relevant to this specific model's goals (e.g. positional information for the carrier embeddings) would allow us to more closely associate the synonyms with the target homonym and directly compare the way that these terms compose with the context. 

In DXA and SER, the target tokens attended over every source token equally. A training paradigm that introduces informed sparsity can reduce the confusion that %words such as determiners and conjunctions might introduce.
stop words may introduce. % Future work might explore the effects of increasing dropout in training.

\subsection{Data Informativeness}
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/freq_by_group.png}
    \caption{Distribution of model predictions on test set}
    \label{fig:pred_dist}
\end{figure}

While our results tend to surpass the naive similarity approach, the Spearman's correlation shows a weak monotonic relationship. One possible explanation is in how the accuracy is measured. As shown in Figure \ref{fig:pred_dist}, even though within architectures the performance stays consistent, the distributions are vastly different. It indicates that the acceptable range for the cases can be fairly large. One might expect that a stricter measure of accuracy would tie closer to the Spearman's correlation.

Additionally, while WordNet is a quite useful resource for dynamic synonym generation, we cannot guarantee that a given homonym's sense is contained within the word's synsets. We tried to mitigate this by appending [MISC] as a catch-all for any senses not contained in the synsets. This also allows for calculations on homonyms which have \textit{no} synsets, but this final tag is limited in its contentfulness, and ultimately may be superfluous in its current construction.

Inconsistent feature factors, such as variable judged meaning length and WordNet synsets, provides difficulties for padding. Despite accounting for padding in the attention masking, it is unclear how it affects the classifier. Moreover, in SER, this results limits us to a concrete number of senses across all homonyms, limiting our ability to account for variable ambiguity in words. 

% This may imply that DistilBERT is especially robust to the different architectures, but experimenting with more models across them would be beneficial.

%This indicates that how the finetuning we use converges the pretrained model knowledge for the task. Additionally, given the novelty of the task, this shows that we can essentially consider all of the models to not be pretrained at all 

% \section{Appendix}
% \subsection{Similarity Module Scores}

% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/cce_cv_all.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/cce_cv_negative.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/cce_cv_neutral.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/cce_cv_positive.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/cce_mean_all.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}

% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/cce_mean_negative.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}

% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/cce_mean_neutral.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}

% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/cce_mean_positive.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}

% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_cv_all.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_cv_negative.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_cv_neutral.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_cv_positive.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_mean_all.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}

% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_mean_negative.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}

% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_mean_neutral.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_mean_negative.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=1\linewidth]{figures/dxa_mean_positive.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}

% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/ser_cv_negative.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/ser_cv_neutral.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/ser_cv_positive.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/ser_mean_all.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/ser_mean_negative.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/ser_mean_neutral.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}
% \begin{figure}[h!]
%     \centering
%     \includegraphics[width=0.75\linewidth]{figures/ser_mean_positive.pdf}
%     \caption{Enter Caption}
%     \label{fig:placeholder}
% \end{figure}



%\begin{comment}
 %   - i don't remember exactly what was said about the distribution of data/predictions
%    - also the thing about how accuracy is calculated (considering average?)
%    - suggestion : Frame the novelty of the task as specifically oriented towards human intuition as opposed to statistical distribution?
    
%\end{comment}


%     TODO

% -pretraining (span information for SER)\\
%     - cross entropy on positive and negative embedding pairs (GLiNER)
% -FFN training for scoring (0-4/1-5)

% \subsection{Other Possible Shortcomings}

    % TODO
    % size of model???
    % not fine tuning the models directly ULMFiT style? (or maybe this isn't worth considering)
    % general amount of compute available (bottleneck for most things)




% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}

\end{document}
