 % This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{booktabs}
\usepackage{graphicx}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\usepackage{graphicx}

\title{VerbaNexAI Lab at SemEval-2024 Task 1: A Multilayer Artificial Intelligence Model for Semantic Relationship Detection}


  \author{Anderson Morillo \and Daniel Peña\\ \bf{Juan Carlos Martinez-Santos \and Edwin Puertas}
  \\Universidad Tecnologica de Bolivar, Cartagena Colombia \\ 
  \texttt{epuerta@utb.edu.co}}





\begin{document}
    
\maketitle


\begin{abstract}

This paper presents an artificial intelligence model designed to detect semantic relationships in natural language, addressing the challenges of SemEval 2024 Task 1. Our goal is to advance machine understanding of the subtleties of human language through semantic analysis. Using a novel combination of convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and an attention mechanism, our model is trained on the STR-2022 dataset. This approach enhances its ability to detect semantic nuances in different texts. The model achieved an 81.92\% effectiveness rate and ranked 24th in SemEval 2024 Task 1. These results demonstrate its robustness and adaptability in detecting semantic relationships and validate its performance in diverse linguistic contexts. Our work contributes to natural language processing by providing insights into semantic textual relatedness. It sets a benchmark for future research and promises to inspire innovations that could transform digital language processing and interaction.

\end{abstract}


%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

The analysis of semantic relationships in natural language is considered an essential pillar for understanding the inherent complexity of textual communication \citet{wolfe2005causal}. With the increasing application of artificial intelligence (AI) models in natural language processing, the ability to discern semantic similarity between text fragments has become a fundamental challenge due to the complexity of natural language and the diversity of meanings that words and phrases can have in different contexts \citet{zunino2023comprender}. In this context, Semantic Textual Relatedness (STR) is a crucial element in natural language understanding, gaining increasing significance with integrating artificial intelligence (AI) models in language processing. 

This article aligns with the objectives set by SemEval 2024 Task 1, a pivotal challenge centered on predicting semantic textual relationships between sentence pairs in the English language. The task's importance lies in its profound impact on advancing contextual language understanding, a cornerstone for AI applications across diverse domains.

Our approach to tackle this challenge involves a four-layer feature extraction process. The first layer focuses on extracting lexical similarity, providing a foundation for understanding semantic connections based on word usage. Subsequently, the second layer delves into capturing knowledge-oriented similarity and incorporating domain-specific insights into the model. The third layer concentrates on Corpus-oriented features, considering the contextual influence of larger text corpora. Finally, in the fourth layer, we employ an Embedding approach. Here, we train a Long Short-Term Memory (LSTM) model, extracting sentence features from phoneme embeddings and a sentence transformer model, thereby capturing nuanced semantic nuances.

Throughout the SemEval 2024 Task 1 competition \cite{ousidhoum-etal-2024-semeval}, our system secured the 24th position out of 36 teams, achieving a competitive score of 0.8192. Notably, our system was designed and optimized for the English language. To foster transparency and collaboration, we have released our code, accessible at https://github.com/VerbaNexAI/SemEval2024.


%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}

The analysis of semantic relations is considered fundamental for understanding the connection of meanings between words, phrases, and sentences in a text. Various relationships, such as synonymy, antonymy, hyperonymy, meronymy, and cohyponymy, can manifest in this context. Two main approaches have addressed this field: rule-based and machine learning-based approaches.

Rule-based approaches use ontologies, structures that define concepts and their relationships, and semantic networks, which are graphical representations of these relationships. Lexical patterns, which are rules that describe semantic relationships based on the structure of words, have also been used. On the other hand, approaches based on machine learning have gained relevance, utilizing technologies such as convolutional neural networks (CNN), recurrent neural networks (RNN), attention models, and word embeddings.

In the literature, we can find several methods for semantic relation detection. In "Learning short-text semantic similarity with word embeddings and external knowledge sources" \citet{nguyen2019learning}, authors propose an approach that uses word embeddings and external knowledge to measure semantic similarity between short texts, managing to outperform traditional methods on diverse datasets.

Another significant work is "A multi-layer system for semantic relatedness evaluation" \citet{gomaa2019multi}, which presents a multi-layer system for semantic relatedness evaluation between sentences, combining various similarity features and achieving promising accuracy on the SICK dataset.

In addition, "A New Methodology for Computing Semantic Relatedness: Modified Latent Semantic Analysis by Fuzzy Formal Concept Analysis" \citet{jain2020new} proposes a hybrid methodology that combines latent semantic analysis and fuzzy formal concept analysis to compute the semantic relatedness between words and sentences, obtaining improved results compared to other baseline measures on a specific corpus.

In the field of language-specific semantic relatedness detection, "Sentence Embedding and Convolutional Neural Network for Semantic Textual Similarity Detection in Arabic Language" \citet{mahmoud2019sentence} proposes a deep learning-based approach to detect paraphrases in Arabic, using word2vec and a convolutional neural network to overcome traditional methods and other stylometric feature-based approaches.

Finally, "Attention-based model for predicting question relatedness on Stack Overflow" \citet{pei2021attention} introduces a deep learning model called ASIM, which uses the attention mechanism to predict the semantic relationship between questions in programming question and answer websites. This model outperforms previous models in terms of performance and generalization in detecting duplicate questions and predicting the relationship between knowledge units.

Despite these advances, knowledge gaps persist. Most models focus on semantic relationships at the word or phrase level. However, we need more research in sentence- and paragraph-level relationship detection. In addition, we require more robust models to adapt to different domains and text types, ensuring a more complete and accurate understanding of semantic relations in natural language processing.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{System Overview}\label{section:system_overview}

This section outlines our proposed model for tackling the task presented in SemEval 2024, Track A, which involves assessing the semantic relationship between pairs of sentences. Initially, the text data undergoes preprocessing, including separating sentence pairs, followed by training a Long Short-Term Memory (LSTM) model on the training dataset. Subsequently, we extract text features based on a four-layer architecture proposed by \citet{gomaa2019multi}, as illustrated in Figure \ref{fig:sys train}. These layers include word embedding, syntactic relationships, corpus topics, and contextual information.

Additionally, we incorporate novel features to enhance our model's performance. These include:

Senticnet: Utilized to extract the polarity of sentences in the knowledge-oriented layer.
Latent Semantic Indexing (LSI) is employed for the corpus-oriented layer to gain insights into the underlying structure of the text corpus.

Phoneme Extraction: A novel approach to capture phonetic information from the sentences.

Furthermore, we integrate an attention mechanism inspired by \citet{vaswani2017attention} to effectively capture intricate dependencies within sequences. Leveraging insights from recent advancements, our model incorporates a Part-of-Speech (POS)-aware and layer ensemble transformer, further enhancing its ability to discern semantic relationships.

By drawing from diverse studies on data augmentation, ensemble learning, and transformer-based profiling, our model aims to provide a robust solution for semantic relationship detection. It showcases a comprehensive understanding of attention mechanisms and their integration with state-of-the-art techniques.


\begin{figure*}[ht!]
    \centering
    \includegraphics[width=0.85\textwidth]{Figures/semeval 2024}
    \caption{System General Pipeline}
    \label{fig:sys train}
\end{figure*}


\subsection{Data Description}

We used the dataset STR-2022 proposed by \citet{DBLP:journals/corr/abs-2110-04845} and collected by \cite{ousidhoum2024semrel2024} for training the system. This dataset comprises 5,500 pairs of sentences in English. This dataset underwent meticulous curation procedures, sampling sentences from various sources, such as social media tweets, book reviews, and paraphrases, to encompass diverse linguistic characteristics and styles. Each pair is labeled with their relatedness score and distribution as shown in Table \ref{Score frequency}.


\begin{table}[ht]
\centering
\caption{Frequency distribution of scores intervals} 
\begin{tabular}{cc}
\hline
\textbf{Score intervals} & \textbf{Frequencies} \\ \hline
0 - 0.2 & 502 \\
0.2 - 0.4 & 1376 \\ 
0.4 - 0.6 & 1861 \\
0.6 - 0.8 & 1149 \\
0.8 - 1 & 612 \\ \hline
\end{tabular}
\label{Score frequency}
\end{table}


\subsection{Data Preprocessing}

During preprocessing, we separated sentences, and verb and subject decontraction were applied. Subsequently, the model was evaluated with and without lemmatization, as well as with and without stopwords, to assess differences in performance. We removed capitalization, special characters, and numbers as part of the preprocessing process. Data preprocessing is a fundamental step that significantly influences the validity and performance of text classification models, both modern transformers and traditional classifiers \citet{siino2024text}. Several preprocessing decisions, such as the treatment of negation, conversion of text to lowercase, application of hyphenation, and consideration of corpus size and document length, are critical to ensure the capture of the true textual meaning and improve the reliability \citet{hickman2022text}.


\subsection{Training Embedding Models}

This part details the training process of a Long Short-Term Memory (LSTM) neural network model for relatedness identification. It begins with data splitting into training and validation sets, followed by message tokenization and constructing a unique vocabulary. We indexed words and applied padding to standardize sequences. We converted the data into PyTorch tensors and defined a custom dataset and data loaders to handle training batches efficiently.

We defined the model with an embedding layer, an LSTM layer, and a linear output layer. During training, the Adam optimizer and Mean Squared Error (MSE) loss function are utilized, with a loop updating model weights over multiple epochs. For monitoring, we evaluated model performance on the validation set after each epoch.

Finally, upon completion of training, the model and its parameters are saved to a file for future use, enabling its application without the need to retrain it from scratch.


\subsection{Feature Extraction}

In this section, we explain how the new features are built and used within the text extraction; we try to create a system that could receive the pairs of sentences and return values with consistent output shapes that can feed the model.


\subsubsection{Layer 1: String-Oriented Similarity}

We based this feature on text extraction, either the characters or the words. It comprises the best features evaluated by \citet{gomaa2019multi}, Cosine Similarity, Jaccard Similarity, Dice’s Coefficient, Bigram, and Trigram.

This layer computes string-oriented similarity features using the following equations:

\begin{equation}
\text{Cosine Similarity} = \frac{A \cdot B}{\|A\| \|B\|}
\end{equation}

\begin{equation}
\text{Jaccard Similarity} = \frac{|A \cap B|}{|A \cup B|}
\end{equation}

\begin{equation}
\text{Dice’s Coefficient} = \frac{2|A \cap B|}{|A| + |B|}
\end{equation}

where \(A\) and \(B\) represent the sets of words in both sentences.


\subsubsection{Layer 2: Corpus-Oriented Similarity}

The system extracts general information using Latent Semantic Indexing (LSI) with the framework Gensim described by \citet{rehurek_lrec} to capture the overall thematic similarity between two sentences. 

We extract the representation of the average LSI values for the primary five topics, aiming to capture the thematic similarity between the sentences.


\subsubsection{Layer 3: Knowledge-Oriented Similarity}

This layer is oriented to extract semantic information related to sentiment within the sentences. We used SenticNet proposed by \citet{cambria-etal-2022-senticnet}, a commonsense-based Neurosymbolic framework that extracts the polarity of words. LSI was extracted by averaging the polarities of each sentence and creating a vector with it.


\subsubsection{Layer 4: Sentence Embedding}

We proposed three forms of word embedding, taking advantage of the good behavior of this layer to evaluate the semantic relationships within two sentences. We used the pre-trained model of sentence transformer \citet{ni2021large}, LSTM, and the phonemes embedding. The phoneme embedding works by taking each letter within the sentences, extracting its representation to a phoneme, and returning its representation as a vector.


\subsection{Classifiers}

We compared the performance of various machine learning models proposed in \cite{gomaa2019multi} and evaluated the combination of different characteristic types represented by vector inputs. 

Random Forest, Gradient Boosting, Multi-layer Perceptron, AdaBoost, and Support Vector Regression (SVR) were employed using the framework Sklearn proposed by \citet{scikit-learn}, alongside an ensemble voting system combining them. This research aimed to identify the most suitable model(s) for analyzing diverse feature vector inputs.


\subsection{Evaluation}

The evaluating part of the code serves as a fundamental component within an empirical study focused on assessing the performance of machine learning classifiers. Its primary purpose is to automate the evaluation process, enabling the systematic comparison of various classifiers in a supervised learning context. By implementing k-fold cross-validation, the code ensures robustness and reliability in the evaluation by mitigating potential biases associated with a single train-test split.

Within the evaluating part, we compute a comprehensive suite of performance metrics for each classifier, including Spearman correlation, Mean Squared Error (MSE), R-squared (R²), Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Mean Absolute Percentage Error (MAPE). These metrics provide a multifaceted assessment of the classifiers' predictive capabilities, accuracy, and robustness.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experimental Setup} \label{section:experimental_setup}

The system model utilized sentences without lemmatization and stopword removal, preserving the original form of the sentences to capture a broader range of semantic nuances. Phoneme extraction, as proposed by \citeauthor{del2023analisis}, was incorporated into the system, utilizing specific components of the provided code. This inclusion aims to enhance the model's sensitivity to phonetic features, contributing to a more comprehensive understanding of textual relationships.

We employ the ShuffleSplit method for cross-validation to assess the model's training. The dataset was split into training and validation sets using a test size 25\%, and we utilized ten pairs (n\_splits = 10) to ensure robust evaluation. We set the random state to 42 (random\_state = 42) for reproducibility. The details of the library versions used in the implementation are provided in Table \ref{tab:python-libraries-versions}.


\begin{table}[htbp]
  \centering
  \caption{Python Libraries and Versions}
  \label{tab:python-libraries-versions}
  \begin{tabular}{lc}
    \toprule
    \textbf{Library} & \textbf{Version} \\
    \midrule
    nltk & 3.8.1 \\
    gensim & 4.3.2 \\
    spacy & 3.7.2 \\
    scikit-learn & 1.3.2 \\
    sentence-transformers & 2.2.2 \\
    senticnet & 1.6 \\
    numpy & 1.23.4 \\
    scipy & 1.10.1 \\
    matplotlib & 3.7.4 \\
    seaborn & 0.13.0 \\
    torch & 1.6.0 \\
    pandas & 2.0.3 \\
    epitran & 1.24 \\
    \bottomrule
  \end{tabular}
\end{table}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}\label{section:results}

We evaluated the semantic relation detection model using the training and test datasets, and the results are in Table \ref{results_of_competitions}. Our model secured the 24th position in the competition ranking, achieving a Spearman correlation coefficient of 0.8192 on the English language dataset. It is relevant to note that this is slightly below the baseline value of 0.83 set as baseline.


\begin{table}[ht!]
\centering
\caption{Ranking of results in framing detection classification}
\begin{tabular}{cc}
\hline
\textbf{Lang} & \textbf{Spearman Correlation} \\ \hline
EN   & 0.8192  \\ \hline
\end{tabular}
\label{results_of_competitions}
\end{table}


In addition, Table \ref{tab:Correlation of Spearmans} summarizes the performance of the voting system's configuration, and we also present its performance compared to the best individual model.


\begin{table*}[ht!]
\centering
\caption{Correlation of Spearman's Rank between Various Text Preprocessing Methods and Machine Learning Models}
\label{tab:Correlation of Spearmans}
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}ccc@{}}
\toprule
\textbf{Preprocessing} & \textbf{Machine Learning
Model} & \textbf{Spearman's Correlation Coefficient} \\ \midrule
No lemmatized, no stopwords & AdaBoost  & 0.82 \\
Lemmatized, no stopwords & AdaBoost  & 0.82 \\
No lemmatized, no stopwords & Gradient Boosting  & 0.82 \\
Lemmatized, no stopwords & Gradient Boosting  & 0.82 \\
No lemmatized, no stopwords & Multi-layer Perceptron  & 0.82 \\
Lemmatized, no stopwords & Multi-layer Perceptron  & 0.82 \\
No lemmatized, no stopwords & Voting  & 0.81 \\
Lemmatized, no stopwords & Voting  & 0.81 \\
No lemmatized, stopwords & Multi-layer Perceptron  & 0.77 \\
Lemmatized, stopwords & AdaBoost  & 0.76
\\ \hline
\end{tabular}
}
\end{table*}


While the results obtained are below the established baseline, we recognize opportunities for improvement. Abstraction analysis could reveal the specific contributions of each system component to this performance and guide future improvements. It is also crucial to consider the nature of the baseline and the inherent complexity of the task at hand.


%%%%%%%%%%%%%%%%%%%%%
\section{Limitations}

While presenting our approach for evaluating semantic relations between sentences, it's crucial to acknowledge certain limitations that may impact the interpretation and applicability of our proposed model. We outline these limitations below:

\begin{itemize}
\item Dataset Representativeness: The STR-2022 dataset, comprising 5,500 English sentence pairs, may not fully capture linguistic diversity and semantic nuances across different languages, limiting the model's generalization to diverse linguistic contexts.

\item Preprocessing Impact: Decisions done during preprocessing (such as removing capital letters, special characters, and numbers) could significantly affect semantic representations. When modifying these preprocessing steps, careful consideration is needed to avoid potential bias or information loss.

\item Hyperparameter Sensitivity: The model's performance is sensitive to hyperparameter choices, like the number of LSTM layers or the learning rate of the Adam optimizer. Fine-tuning is crucial for optimizing the model's ability to capture semantic relationships effectively.

\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Ethical Considerations}

We linked the text similarity field to the detection of paraphrasing \citep{Mahmoud1}, which can pose an ethical problem when using an author's work without proper citation. Our solution addresses the bias by extracting various text features, from word information to context and vector representation. This way, we can avoid some limitations from training the model with insufficient features.


%%%%%%%%%%%%%%%%%%%%
\section{Conclusions}

This paper presented a comprehensive approach to evaluating semantic relations between sentences, addressing the challenges posed by SemEval 2024 Task 1. Our model employs a sophisticated four-layered feature extraction technique, encompassing lexical similarity, knowledge orientation, corpus orientation, and embedding layers.

Despite achieving a notable 24th place in the competition, we acknowledge certain limitations, including concerns about dataset representativeness, preprocessing decisions, and hyperparameter sensitivity. These insights serve as valuable lessons for future enhancements in our approach.

While the Spearman correlation of 0.8192 places our model slightly below the established baseline of 0.83, this outcome provides an invaluable learning experience. Moving forward, we plan to conduct ablation studies to dissect the impact of individual components, explore alternative models and preprocessing strategies, and conduct a detailed error analysis to address specific shortcomings.

Ultimately, this work contributes to a deeper understanding of semantic relations and provides a competitive model for SemEval 2024. We are committed to advancing semantic understanding and improving AI systems for natural language processing. The journey from this competition is a stepping stone toward more refined and practical solutions in semantic relationship detection.


%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Acknowledgments}

To the SemEval contest, sponsored by the SIGLEX Special Interest Group on the Lexicon of the Association for Computational Linguistics. To the master's degree scholarship program in engineering at the Universidad Tecnológica de Bolivar (UTB) in Cartagena, Colombia.

We would like to express our gratitude to the team at the VerbaNex AI Lab \footnote{https://github.com/VerbaNexAI} for their dedication, collaboration, and ongoing support of our research endeavors. 

\bibliography{custom}

\end{document}
