%
% File acl2015.tex
%
% Contact: car@ir.hit.edu.cn, gdzhou@suda.edu.cn
%%
%% Based on the style files for ACL-2014, which were, in turn,
%% Based on the style files for ACL-2013, which were, in turn,
%% Based on the style files for ACL-2012, which were, in turn,
%% based on the style files for ACL-2011, which were, in turn, 
%% based on the style files for ACL-2010, which were, in turn, 
%% based on the style files for ACL-IJCNLP-2009, which were, in turn,
%% based on the style files for EACL-2009 and IJCNLP-2008...

%% Based on the style files for EACL 2006 by 
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith

\documentclass[11pt]{article}
\usepackage{acl2023}
\usepackage{times}
\usepackage{url}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{color}
\usepackage{natbib}
\usepackage{graphicx}
\usepackage{float}
\setlength\titlebox{7cm}

% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.


\title{GIL-IIMAS UNAM at SemEval-2024 Task 1: \\ SAND: An In Depth Analysis of Semantic Relatedness Using Regression and Similarity Characteristics.}

\author{F. López-Ponce$^{1}$, Ángel Cadena$^{1,2}$, K. Salas-Jimenez$^{1,2}$, \\ \textbf{D. Preciado Márquez$^{1}$, G. Bel-Enguix$^{1,3}$} \\
  $^{1}$Grupo de Ingeniería Lingüística - UNAM\\
  $^{2}$Posgrado en Ciencias e Ingeniería de la Computación - UNAM \\
    $^{3}$Departament de Filologia Catalana i Lingüística General - Universitat de Barcelona \\
 \texttt{
 \{francisco.lopez.ponce,karla\_dsj\}@ciencias.unam.mx} \\ 
 \texttt{\{angelcaden,davidpreciado115\}@gmail.com} \\ %
 \texttt{gbele@iingen.unam.mx}}

\date{}

\begin{document}
\maketitle
\begin{abstract}
  The STR shared task aims at detecting the degree of semantic relatedness between sentence pairs in multiple languages. Semantic relatedness relies on elements such as topic similarity, point of view agreement, entailment, and even human intuition, making it a broader field than sentence similarity. The \textbf{GIL-IIMAS UNAM} team proposes a model based in the SAND characteristics composition (Sentence Transformers, AnglE Embeddings, N-grams, Sentence Length Difference coefficient) and classical regression algorithms. This model achieves a 0.83 Spearman Correlation score in the English test, and a 0.73 in the Spanish counterpart, finishing just above the SemEval baseline in English, and second place in Spanish.
\end{abstract}

\section{Introduction}

The Semantic Textual Relatedness (STR) task \cite{ousidhoum-etal-2024-semeval} aims at creating systems that measure STR on pairs of sentences based on their closeness in meaning \citep{abdalla2023makes}. This task is comprised of three tracks. Tracks A and B focus on monolingual models. Track A only accepts supervised models trained with the available tagged datasets, whereas track B focuses on the unsupervised approach relying on the same datasets but without the tagged score. Track C is the cross-lingual case where the target language has to follow an unsupervised approach. The datasets provided consist of sentence pairs that were sampled from various semantic similarity datasets. 

%\textcolor{red}{Each sentence ranges from 5 to 25 words, and the corresponding score for each pair ranges from 0 (maximally unrelated) to 1 (maximally related). This score was manually annotated and agreed upon the annotators}. 
This task expands upon classic sentence similarity comparisons, encouraging the use models and algorithms capable of analyzing more than meaning of a pair of sentences, focusing deeper on characteristics such as the syntactic structure of the sentences, lexicon relationships, as well as meaning and emotion. The GIL-IIMAS UNAM team participated in Track A.
Although it included nine languages we have only worked with the Spanish and English dataset.

Track A is a regression problem since each dataset contains the sentence pairs as well as a corresponding sentence relatedness score that ranges from 0 to 1. The evaluation compares the ground values in the test set with the proposed model's prediction, meaning track A is evaluated using the Spearman Correlation.

This paper makes use of the SAND composition, a set of STR relevant characteristics, as well as regression algorithms trained with these characteristics in order to predict the STR score of other sentence pairs. In this paper the precise characteristics, algorithms, and parameters are presented as well as language based analysis. The final scores correspond to the best performing regression algorithm.


Our work also compares different metrics and their influence on the STR task compared to classic semantic similarity, as well as the model's varying behavior over the different languages used.

The paper is structured as follows: Section \ref{relatedWork} explains the theories and models that are the background of our proposal. Section \ref{Sand} explains the set of characteristics that have been chosen for our experiments. Section \ref{expSetup} explains the configuration of the data and the experimental methodology. In Section \ref{results} we discuss the results obtained in the experiments, and compare them to the scores of other participants in the track. We close in Section \ref{conclusions} with some conclusions and ideas for further experiments. 

\section{Related Work}\label{relatedWork}

In the field of STR, various methodologies have been proposed. One such approach, outlined by \citet{asaadi-etal-2019-big}, involves analyzing the relatedness between word bi-grams. They describe the construction of a dataset tailored for this purpose. To compute the STR between bi-grams, or between bi-grams and unigrams, they utilize word embeddings represented as vectors generated by GloVe, fastText, and models employing matrix factorization of word-context co-occurrence matrices. They explore various methods for composing bi-gram vectors, such as addition, multiplication, tensor product with convolution, and dilation. The relatedness between two vectors is determined by computing the cosine similarity between them.

In a study by  \citet{abdalla2023makes}, the concept of Semantic Textual Relatedness (STR) is extended to encompass the comparison between entire sentences. The authors delineate the construction of a specialized dataset tailored for this task and show the annotation process applied to this dataset. Their investigation delves into the influence of various linguistic factors, including lexical overlap, related words, related words belonging to the same part of speech, and the relatedness of subjects or objects, on the semantic relatedness between pairs of sentences. They represent each sentence as a vector and employ cosine similarity between these vectors as a metric for predicting semantic relatedness. To facilitate this analysis, they use static word embeddings such as Word2Vec, GloVe, and fastText, as well as contextual word embeddings like BERT and RoBERTa.


\section{SAND Composition}\label{Sand}


This section describes the SAND (named based on the used characteristics: \textbf{S}entence Transformers, \textbf{A}nglE Embeddings, \textbf{N}-grams, Sentence Length \textbf{D}ifference coefficient) regression system used for the task. The STR datasets are comprised of senbtence pairs and a target score, in order to train the model with task relevant information certain similarity metrics were chosen in order to create a vector of characteristics that represent each sentence pair, such vectors were used as training data for the regression algorithms. An initial approach relied on similarity metrics such as Jaccard, and Dice coefficients as well as Jaro-Winkler and Levenshtein distance, nonetheless these metrics proved to be inefficient at training the model adequately, returning poor results when evaluated with partitioned training data. After revising the dataset and the nature of the sentences in question, the initial chosen characteristics were a coefficient analyzing the length of the sentences with and without stoptwords. Consider $x, y$ two sentences, then both coefficients are obtained from the following:
\begin{equation}
    \text{LenCoef}(x,y) = |\frac{\text{length}(x) - \text{length}(y)}{\text{length}(x) + \text{length}(y)}|
\end{equation}

An observation on lexical overlapping in highly related sentences led to the choice of an n-gram based coefficient. For $n \in \{1, 2, 3\}$, the n-gram coefficient is defined as:

\begin{equation}
    \text{n-gramCoef}(x,y) = \frac{\text{n-grams}(x) \cap \text{n-grams}(y)}{\text{n-grams}(x)}
\end{equation}

Sentence pairs in the dataset often rely on contextual similarity apart from lexical overlap when it comes to measuring relatedness, meaning the use of a pretrained model is in order. The first approach relied on the use of Sentence Transformers (ST) \citep{reimers-2019-sentence-bert}, a siamese neural net that uses pretrained encoders in order to generate contextualized sentence embeddings. Each pair of sentences was embedded using the ST architecture and the cosine similarity of the resulting vectors was obtained as the initial pretrained characteristic. Formally speaking:

\begin{equation}\label{cosine_sim}
    \text{ST}_{sim}(x,y) = \cos (\theta ) =   \dfrac {x \cdot y} {\left\| x\right\| \left\| y\right\|}  
\end{equation}

Nonetheless standard encoder generated embeddings have shown to be improvable, for that the final characteristic relies on similarity based on AnglE-optimized embeddings \citep{li2023angle}. These type of embeddings optimize the cosine similarity saturation zones during training using complex space embeddings so that resulting vectors achieve a higher level of similarity. Nonetheless the final comparison between AnglE vectors is done in the same manner as equation \ref{cosine_sim}.

Once these characteristics are extracted from each sentence pair they are passed to various regression algorithms for training, validation and testing. For this task four regression algorithms were used. The reported scores correspond to the best model for each language. The precise details of the implementation are presented next.

\section{Experimental Setup} \label{expSetup}

\subsection{Data and Evaluation Methodology}
We use the official dataset \cite{ousidhoum2024semrel2024} in English and Spanish for Task 1, track A (supervised), which is structured as follows: PairID, Text and Score. PairID is an identifier of the pair, the Text column is the sentence pair separated by a line break, and the Score column is a float number between 0 (completely unrelated) and 1 (maximally related) which indicates the degree of semantic textual relatedness between the two sentences. 

The English training corpus is composed of 5500 sentence pairs meanwhile the Spanish counterpart has 1561 pairs. The score distribution comparison in figures \ref{distribucion_scores_english} and \ref{distribucion_scores_spanish} indicates that the English scores have a wider variance than the Spanish ones, nonetheless the most represented scores (scores corresponding to over 50 sentence pairs) follow roughly a Gaussian distribution. In contrast the Spanish score distribution doesn't behave as cleanly.

\begin{figure}[H]
\includegraphics[scale=0.35]{data_eng.JPG}
\centering
\caption{English score distributions.}
\label{distribucion_scores_english}
\end{figure}
\begin{figure}[H]
\includegraphics[scale=0.5]{data_esp.PNG}
\centering
\caption{Spanish score distributions.}
\label{distribucion_scores_spanish}
\end{figure}

The results of the shared task are evaluated with the Spearman rank correlation coefficient which is used to discover the strength of a relationship between two sets of data, in this case two sentences.


\subsection{Algorithms, parameters and pretrained models}

%CREO QUE PODEMOS QUITAR ESTE PARRAFO 
%The following libraries were used to extract the SAND characteristics: pandas (load the dataset and add a column for each characteristic for classification), nltk (clean stopwords from dataset), sklearn (import the function for classification and extracting n-grams), transformers, huggingface\_hub and angle\_emb (these last three make it possible to create bert and angle embeddings).

The following regression algorithms, along with the particular parameters, were trained and validated using 7-dimensional vectors corresponding to each characteristic in the SAND Composition:
\begin{itemize}
    \item \underline{SVM} and \underline{SVM with epsilon}:  Both algorithms' used the default regularization parameter of 1, meanwhile the SVM $\epsilon$ variation used $\epsilon = 0.3$.
    \item \underline{RandomForest}: Default parameters were used except for maximum depth which was set to 5, and randomness of the bootstrapping of the samples used when building trees was set at 0.
    \item \underline{Ridge regression}: Default parameters except the constant that multiplies the L2 term $\alpha$. For the English corpus $\alpha$ was set at 0.8, while for the Spanish corpus it was set at 0.9.
\end{itemize}
Regarding ST and AnglE Embeddings the use of a pretrained model was necessary in order to compute the embeddings and eventually the similarity score. For the English version of ST the \textsc{all-mpnet-base-v2} checkpoint was used, meanwhile for the Spanish version the \textsc{paraphrase-multilingual-mpnet-base-v2} checkpoint was chosen, both developed particularly for the ST architecture \cite{reimers-monomultilingual}. Meanwhile AnglE embeddings used the \textsc{angle-bert-base-uncasednli-en-v1} checkpoint, prioritizing the BERT \cite{devlin-etal-2019-bert} based model over the LLaMA \cite{touvron2023llama} based one due to computational power needed for each model.

\section{Results and Analysis} \label{results}

The final SAND composition was the result of an ablation test performed using various combinations of different characteristics. The previously mentioned regression algorithms were trained and tested using each individual characteristic as well as different combinations of each. The SAND composition was the best performing combination for both English and Spanish, each of the four regression algorithms achieved the best result in the ablation test with SAND than with each simpler combination. 


Tables \ref{Ablation_English} and \ref{Ablation_Spanish} show the best Spearman Correlation for each characteristic combination as well as the model that achieved said score in each language when evaluating in the validation dataset.

\begin{table}[htbp!]
    \centering
    \begin{tabular}{|c|c|c|c|c|}
        \hline 
        \textbf{Char}& \textbf{SVM} & \textbf{RF} &  \textbf{SVM\_{$\epsilon$}} & \textbf{Ridge} \\ \hline
        ST &  0.7891 & 0.7847 & 0.7865 & 0.7891\\ \hline
        AnglE & 0.7789 & 0.7737 & 0.7772 & 0.7789 \\ \hline
        N-grams & 0.6634 & 0.6496 & 0.6620 & 0.6584 \\ \hline
        Distance & 0.2343 & 0.2090 & 0.2839 & 0.2888 \\ \hline
        SAND & \textbf{0.7986} & \textbf{0.8197} & \textbf{0.7992} &  \textbf{0.7921} \\ \hline
    \end{tabular}
    \caption{English Spearman Correlation for different characteristics.}
    \label{Ablation_English}
\end{table}

\begin{table}[htbp!]
    \centering
    \begin{tabular}{|c|c|c|c|c|}
        \hline 
        \textbf{Char} & \textbf{SVM} & \textbf{RF} &  \textbf{SVM\_{$\epsilon$}} & \textbf{Ridge} \\ \hline
        ST& 0.6397 & 0.6058 & 0.6310 & 0.6397\\ \hline
        AnglE & 0.6140 & 0.5976 & 0.6038 & 0.6212\\ \hline
        N-grams &  0.6425 & 0.6232 & 0.6419 & 0.6431\\ \hline
        Distance &  0.5595 & 0.5584 & 0.5586 & 0.5594\\ \hline %quiero creer
        SAND & \textbf{0.688} & 0\textbf{0.6783} & \textbf{0.6937} & \textbf{0.7029}\\ \hline
        %Angle + SBERT & 0.8146 \\ \hline
        %\{1,2\}-Grams+ Distance &  0.6674     \\ \hline 
        %\{1,2,3\}-Grams+ Distance &  0.6655 \\ \hline 
        %\{1,2,3\}-Grams & 0.6634  \\ \hline 
        %\{1,3\}-Grams+ Distance &  0.6627    \\ \hline 
        %\{2,3\}-Grams+ Distance &  0.57    \\ \hline 
        %Embs + unigram & 0.8152  \\ \hline 
        %Embs + bigram & 0.8154  \\ \hline 
        %Embs + trigrams & 0.8151  \\ \hline 
        %Embs + {1,2}-Gram: & 0.8171   \\ \hline 
        %Embs + Distance &  0.8119  \\ \hline 
        %Embs + {1,2,3}-Gram &  0.8164  \\ \hline 
    \end{tabular}
    \caption{Spanish Spearman Correlation for different characteristics.}
    \label{Ablation_Spanish}
\end{table}

Finally table \ref{SANDResults} shows the results of each model with the SAND Composition. The reported results correspond to the predicted scores made by the highlighted models: Random Forest for English, and Ridge Regression for Spanish.

\begin{table}[htbp!]
    \centering
    \begin{tabular}{|c|c|c|}
    \hline
    \textbf{Model} & \textbf{Spanish} & \textbf{English} \\ \hline
    RandomForest & 0.6968 & \textbf{0.8197} \\ \hline
    SVM & 0.6881 & 0.8133 \\ \hline
    Ridge & \textbf{0.7029} & 0.8117 \\ \hline
    SVM Epsilon & 0.6997 & 0.7945 \\ \hline
    \end{tabular}
    \caption{Spearman coefficient for SAND composition.}
    \label{SANDResults}
\end{table}

It is important to note that the embeddings created with pretrained models were the feature with the greatest impact on our model. Even as an isolated measure they both prove to be better metrics than their n-grams and distance counterparts. Similarly, considering that both \textsc{BERT} and \textsc{all-mpnet-base-v2} are trained in English primarily, it is logical that the regression algorithms performed better in said language.

Nonetheless the SAND Composition proves that these characteristics can be complemented and improved using relevant information such as n-grams coefficients. Since they don't rely on pretrained models and rather focus on lexical overlapping, this coefficient was able to discern certain relatedness measures.

SAND Composition was able to achieve the best results of the ablation test meaning that regression models do benefit from the mix of characteristics and still be relevant in a competition setting.

\section{Conclusion} \label{conclusions}
In this paper, we describe the SAND composition for the STR shared task, which is based on both semantic and lexical features, because we observe that: two sentences can share most of the words and apparently have no semantic relation but a high value of Spearman coefficient and vice versa, they can share semantics without matching words. With this in mind the SAND Composition contains half semantic features, and half lexical ones. This approach allowed achieved the 18th place in English and 2nd place in Spanish, with 0.83 and a 0.73 Spearman Correlation score respectively. In both cases the results are over the baseline and only 0.05 of the first place in English and 0.01 in Spanish, meaning SAND proves to have relevant characteristics.

For future experiments added features that consider both semantic, lexical and  contextual parts simultaneously might prove to be more efficient than various unrelated metrics. Mixing word embeddings and PoST tagging might generate a relatedness score that proves to be more useful than separate similarity metrics. 


\section*{Acknowledgments}

This research was funded by CONAHCYT (CF-2023-G-64) and PAPIIT project IT100822. G.B.E. is supported by a
grant for the requalification of the Spanish university
system from the Ministry of Universities of the Government of Spain, financed by the European Union,
NextGeneration EU (María Zambrano program, Universitat de Barcelona).

K. Salas-Jimenez thanks CONAHCYT scholarship program (CVU: 1291359).

%KS agradece el apoyo al 
Ángel Cadena thanks CONAHCYT scholarship program (CVU: 1227093).
% include your own bib file like this:
\bibliography{bib-acl}
\bibliographystyle{acl_natbib}


\end{document}