
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt,table,xcdraw]{article}

% Remove the "review" option to generate the final version.
%\usepackage[review]{ACL2023}
\usepackage{ACL2023}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
\usepackage{multirow}
\usepackage{xcolor}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}


% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{OZemi at SemEval-2024 Task 1: A Simplistic Approach to Textual Relatedness Evaluation Using Transformers and Machine Translation}

% Author information can be set in various styles:
% For several authors from the same institution:
\author{Hidetsune Takahashi \and Xingru Lu \\   {\bf Sean Ishijima} \and {\bf Deokgyu Seo} \and {\bf Yongju Kim} \and {\bf Sehoon Park} \and {\bf Min Song}\\ \and {\bf Kathylene Marante} \and {\bf Keitaro-Luke Iso} \and {\bf Hirotaka Tokura}\\ \and {\bf Emily Ohman}\\
        Waseda University \\ ohman@waseda.jp} %\\ Nishiwaseda 1-6-1, Tokyo, JAPAN}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

% \author{First Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\\And
%   Second Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\}

\begin{document}
\maketitle
\begin{abstract}
In this system paper for SemEval-2024 Task 1 subtask A, we present our approach to evaluating the semantic relatedness of sentence pairs in nine languages. We use a mix of statistical methods combined with fine-tuned BERT transformer models for English and use the same model and machine-translated data for the other languages. This simplistic approach shows consistently reliable scores and achieves middle-of-the-pack ranks in most languages.
\end{abstract}

%4 pages!
\section{Introduction}
%Understanding semantic relatedness between texts provides prospective development toward our understanding of how we construct meaning in our sentences and fills the gap between intended instructions and interpreted messages. Through this deconstruction of natural language, we can find that there is a bonafide application that stretches across different fields and allows us to garner insight into languages’ semantics and use this knowledge to build a bridge between context and its relevance in textual linguistics.

SemEval 2024 Task 1 \citep{semrel2024task} calls for assigning scores indicating semantic textual relatedness (STR) of sentence pairs in 14 different languages. We participate in Track A, which is the supervised subtask for systems that have been trained using the provided labeled datasets \citep{semrel2024dataset}. There are data in Algerian Arabic, Amharic, English, Hausa, Kinyarwanda, Marathi, Moroccan Arabic, Spanish, and Telugu for Track A and we provide a solution for all 9 languages. The labeled data has been manually annotated for relatedness using a comprehensive annotation framework \citep{abdalla-etal-2023-makes}. %Using this data, developed systems are applied to various sentence pairs to accurately rank their closeness in meaning across 14 languages. As A

A large portion of previous work in STR has been conducted for English-language data. This task does include English, but the focus is on lower-resourced languages \citep{hedderich2021survey,marreddy2022resource}. STR is a crucial component in information retrieval, summarization, and question answering, as well as in developing Large Language Models (LLMs). The lack of STR or similar NLP resources for low-resource languages means progress is often much slower in related research such as the development of LLMs too making the progress achieved through this task societally highly impactful by providing new tools and datasets for language where NLP resources are lacking \citep{vulic2020probing,zhang2020semantics}. %can find similar access to studies that stimulate the growth of STR research in their native tongue, and contribute to the development of the field as a whole. %will add citations(Sean Ishijima) 
% Data



% The Semantic Textual Relatedness Shared Task 1

% The task consists of predicting semantic textual relatedness (STR) of sentence pairs. Participants will rank sentence pairs by their closeness in meaning (i.e., their degree of semantic relatedness) in the 14 different languages. All sentence pairs will have manually determined relatedness scores between 0 (completely unrelated) and 1 (maximally related). Each team can provide submissions for one, two or all of the tracks shown below:

% Track A: Supervised

% Participants are to submit systems that have been trained using the labeled training datasets provided. Participating teams are allowed to use any publicly available datasets (e.g., other relatedness and similarity datasets or datasets in any other languages). However, they must report additional data they used, and ideally report how impactful each resource was on the final results.

% In this paper, we present our approaches for SemEval2024 Task 1 for measuring textual relatedness in a number of low-resource languages such as Hausa, Kinyarwanda, and Telugu % LIST
% and some higher-resource languages, such as English, Spanish, and Arabic%LIST
% .
% The task goal is to develop and evaluate models that accurately determine the similarity between pairs of texts, in this case roughly at sentence-level. %A FEW MORE SENTENCES ABOUT THE TASK

%Semantic similarity has wide applications in fields like information retrieval and machine translation and is a crucial component in detecting the same information phrased differently. %MAYBE MOVE

Our methodology uses both traditional TF-IDF vectorization and transformer models like BERT \citep{devlin2018bert} fine-tuned for semantic relatedness tasks. 
%Our module, designed for versatility and scalability, unites TF-IDF and BERT to offer a comprehensive assessment of textual similarity. 
We leverage the high availability of resources that exist for English to fine-tune a BERT model that we then use on machine-translated versions of the datasets for the other languages (except for Spanish where a multilingual BERT model yielded better results than with machine translating the data).
%We focused on the English task due to the availability of resources, and then for most of the other languages machine-translated the datasets into English to utilize the fine-tuned English model.
This approach seems to capture both lexical patterns and deeper semantic relationships, making it effective for linguistically diverse datasets, and cost-effective because there is no need to manually annotate more than one dataset (language). It is therefore an alternative approach to creating language-specific models. Although our approach is simplistic, it has the upside of working reasonably well for any low-resource language that has some machine translation or parallel language data resources.

%This score not only validates our methodology but also highlights the potential of blending traditional and contemporary NLP techniques.  % Seo Deokgyu 
%A few sentences summarizing the paper.
%What are we doing
%Why are we doing it
%What's the core idea of our approach
%\begin{itemize}
%    \item What is the task about and why is it important? Be sure to mention the language(s) covered and cite the task overview paper. ~1 paragraph



%\item What is the main strategy your system uses? ~1 paragraph


\section{Background}

In SemEval-2024 Task 1, the dataset was adapted from the STR-2022 dataset \citep{abdalla-etal-2023-makes}. The STR-2022 dataset contains 5,500 English sentence pairs that were manually annotated using a comparative annotation framework, yielding fine-grained scores ranging from 0 to 1 (maximally unrelated to maximally related). The dataset was constructed by sampling sentences from various sources to capture a wide range of text characteristics such as sentence structure, formality, and grammaticality. The sources include datasets on formality \citep{rao2018dear}, book reviews \citep{wan2018}, paraphrases \citep{wieting-gimpel-2018-paranmt}, natural language inference \citep{bowman2015large}, semantic textual similarity \citep{cer2017semeval}, stance \citep{mohammad2016dataset}, and text simplification \citep{horn2014learning}.
% \begin{enumerate}
%     \item Formality pairs with the same meaning but different levels of informality \citep{rao2018dear}
%     \item Goodreads book reviews \citep{wan2018} 
%     \item ParaNMT paraphrases from a machine translation system \citep{wieting-gimpel-2018-paranmt}
%     \item SNLI pairs of premises and hypotheses for natural language inference \citep{bowman2015large}
%     \item STS pairs of sentences with semantic similarity scores \citep{cer2017semeval}
%     \item Stance tweets labeled for sentiment and stance towards specific targets \citep{mohammad2016dataset}
%     \item Wikipedia Text Simplification Dataset pairs of Wikipedia sentences and their simplified forms \citep{horn2014learning}
% \end{enumerate}

The corresponding datasets for the other languages are much smaller and consist of roughly 1000 sentence pairs each with minor variations in size.

%How is semantic relatedness different from semantic similarity? %Luke
%Before progressing with this task, the difference between ‘semantic similarity’ and ‘semantic relatedness’ must be clarified. 
Semantic \textbf{relatedness} and semantic \textbf{similarity} are closely related concepts in natural language processing (NLP), however, the terms are not interchangeable. Semantic similarity is a narrower definition that only takes term similarity into account (e.g. \textit{fork} is similar to \textit{knife}), whereas relatedness in addition to similarity can include terms or concepts that are related beyond hyponymic relationships such as \textit{fork} being related to \textit{eating}) \citep{asaadi2019big,BATET2016365}.  % used as each of them indicates a distinct aspect of the relationship between words or concepts. \citet{BATET2016365} outline that semantic similarity is the level of taxonomic similitude between two textual terms or concepts, while semantic relatedness captures the semantic resemblance between these both in terms of their taxonomic and non-taxonomic relationships. In other words, semantic relatedness is a general measure that gauges the strength of the relationship between two terms or concepts \citep{asaadi2019big}. %It considers the contextual, conceptual, or associative connections between two terms, even if they do not share the same definitions. For example, it could be said that “car” and “road” are semantically related as they are often associated with each other despite their meanings not being identical. On the other hand, semantic similarity is more specific as the focus is on measuring how close two words or concepts are in terms of meaning. In other words, it looks at the resemblance or how likely words or phrases are related to each other based on shared semantic features. For instance, “car” and “automobile” are semantically similar as they both refer to the same concept. %[Could someone clarify this? Thanks. Luke] 
This task focuses on the broader concept of relatedness but utilizes more narrowly defined datasets based on similarity as well in the construction of the datasets. 

%In the confines of this paper and task, we are attempting to automatically detect relatedness, but use the term similarity and relatedness interchangeably in text. %something like that? <-- Yes! Thanks!
%While semantic similarity concentrates on how much meaning is shared, semantic relatedness accentuates more expansive associations between concepts. When it comes to utilizing natural language processing, the concepts are significant based on the situation and the context. Therefore, one natural language processing application might require semantic similarity approach and the other might require semantic relatedness approach.%This paper will focus on determining the semantic [insert which one here + why we chose to do so] 

%Previous work on semantic similarity and relatedness has shown that syntax is an important part of similarity detection \citep{wei2023synwmd}, 

%Before the 2024 SemEval competition, the STS Benchmark stood as a compilation of English datasets utilized in Semantic Textual Similarity (STS) activities conducted within the context of SemEval between 2012 and 2017 \citep{sts-benchmark}.

% \begin{itemize}
%     \item In your own words, summarize important details about the task setup: kind of input and output (give an example if possible); what datasets were used, including language, genre, and size. If there were multiple tracks, say which you participated in.

% \item Here or in other sections, cite related work that will help the reader to understand your contribution and what aspects of it are novel.

%\end{itemize}
In recent years the development of NLP resources for low-resource languages has been speeding up, but there are still large discrepancies in what types of tools, models, and resources exist for languages other than English \citep{hedderich2021survey}. There are also significant differences in the resources available among low-resource languages and what being a low-resource language entails \citep{hamalainen2021endangered,marreddy2022resource}. For most of the languages in this task, there are at least some models and tools \cite[see e.g][]{deode2023l3cube} but a handful of research groups working on a language is quite different from nearly all research groups in the world working on producing models and tools for a language (English). When there is a need for more data, often data augmentation methods are used to increase data points. Machine translation is an established method of data augmentation, particularly with low-resource languages where it might not be possible to use language-specific models \citep{amjad2020data}.



\section{System overview}
% Each instance in the training, development, and test sets is a sentence pair. The instance is labeled with a score representing the degree of semantic textual relatedness between the two sentences. The scores can range from 0 (maximally unrelated) to 1 (maximally related). These gold label scores have been determined through manual annotation. Specifically, a comparative annotation approach was used to avoid known limitations of traditional rating scale annotation methods. This comparative annotation process (which avoids several biases of traditional rating scales) led to a high reliability of the final relatedness rankings. Further details about the task, the method of data annotation, how STR is different from semantic textual similarity, applications of semantic textual relatedness, etc. can be found in this paper.

Our choice of methodology was shaped by pedagogical considerations as well as technical. As we participated in this task as part of an undergraduate senior research seminar in computational methods, we purposely started with the simplest most readily available tools progressing towards more advanced methods. Along the way, we compared the results and progress at each step in an attempt to better understand how each of the specific NLP tools worked and how accurate their output was when used on real projects such as this dataset. 

The main strategy of our system is integrating classic NLP methods, such as the Dice Score and TF-IDF, with advanced deep learning techniques like BERT models, to determine semantic relatedness between sentence pairs. Firstly, our system imports a CSV dataset that contains pairs of English sentences (separated by "\textbackslash{n}"), each paired with a relatedness score ranging from 0 to 1. Then, to assess semantic relatedness, the system adopts several basic NLP techniques, including Spacy's Linguistic Features for efficient text processing, TF-IDF for calculating word importance in sentences, Spacy Similarity and Cosine Similarity for measuring sentence similarity, and fine-tuned BERT Models for leveraging contextually rich semantic analysis \citep{devlin2018bert}. These techniques collectively contribute to a robust evaluation of semantic relatedness against the given scores. We tried early on to adopt the same approach to the non-English languages with language-specific transformer-based similarity and relatedness models, but the language-specific models yielded much lower evaluation scores than what the English model achieved with machine-translated versions of the non-English datasets. We used the Google Translate API to translate the datasets into English to maintain consistency in analysis. Compared to other translation APIs such as DeepL, for this task, Google Translate seemed to produce better translations, perhaps because of how it favors more common words over context thus being more suited for STR and/or STS tasks \citep[see e.g.][]{ohman2022self}. %After this process, we adopted the same strategy for translated non-English languages.
%Kim Yongju

%\item What did you discover by participating in this task? Key quantitative and qualitative results, such as how you ranked relative to other teams and what your system struggles with. ~1 paragraph 

Participating in the semantic relatedness task using the hybrid strategy allows for a comprehensive exploration of the system's performance and methodology. Through a detailed analysis, you can assess the effectiveness of traditional NLP methods, including TF-IDF and Spacy's Linguistic Features, in comparison to more advanced deep learning techniques like BERT. Evaluating the impact of contextual embeddings from fine-tuned BERT models provides insights into how well the model captures nuanced semantic relationships. The inclusion of Google Translate for non-English languages offers an opportunity to examine the system's ability to maintain consistency across languages. Assessing the generalization capability, scalability, and efficiency of the system provides a holistic understanding of its applicability to diverse datasets and real-world scenarios. Through this participation, we can uncover strengths, weaknesses, and potential areas for improvement, guiding future research directions and refining the hybrid strategy for enhanced semantic relatedness evaluation across languages and varied linguistic contexts. In particular, this approach shows that it is possible to achieve reasonable accuracies by leveraging the prevalence of tools and models designed for English with low-resource languages. %PARK Sehoon


Our code is available on GitHub \footnote{\url{https://github.com/esohman/SemEval2024}}.





% \begin{table*}[htbp!]
% \centering
% \small
% \begin{tabular}{l|lllll|ll}
%                 & \multicolumn{5}{l}{Train   Data}                                                                             & Dev Data &       \\
% LANGUAGE        & \multicolumn{3}{l}{English Translation}                             & \multicolumn{2}{l}{Multilingual Model} &          &       \\
%  &
%   \begin{tabular}[c]{@{}l@{}}Spacy \\ Similarity\end{tabular} &
%   cos vector &
%   \begin{tabular}[c]{@{}l@{}}fine-tuned \\ SBERT\end{tabular} &
%   DBMCv1\footnote{distiluse-base-multilingual-cased-v1 \citep{sarzynska2021detecting}} &
%   \begin{tabular}[c]{@{}l@{}}all-mpnet-\\ base-v2\footnote{\url{https://huggingface.co/sentence-transformers/all-mpnet-base-v2} and \citet{reimers-2020-multilingual-sentence-bert}}\end{tabular} &
%   \multirow{-2}{*}{Official Score} &
%   \multirow{-2}{*}{Ranking} \\
%   \hrulefill
% Algerian Arabic & 0.25  & 0.44                         & \cellcolor[HTML]{EFEFEF}0.51 & 0.42                         & 0.39    & 0.37    & 18/20 \\
% Amharic         & 0.37  & 0.61                         & \cellcolor[HTML]{EFEFEF}0.78 & 0.16                         & 0.12    & 0.78    &11/16 \\
% English         & 0.34  & 0.61                         & \cellcolor[HTML]{EFEFEF}0.80 & *                            & *       & 0.81    & 10/34 \\
% Hausa           & 0.07  & 0.43                         & \cellcolor[HTML]{EFEFEF}0.65 & 0.21                         & 0.34    & 0.62    & 12/19 \\
% Kinyarwanda     & 0.18  & 0.39                         & \cellcolor[HTML]{EFEFEF}0.57 & 0.3                          & 0.38    & 0.57    & 8/14  \\
% Marathi         & 0.45  & 0.68                         & \cellcolor[HTML]{EFEFEF}0.81 & *                            & *       & 0.86    & 13/25 \\
% Moroccan Arabic & -0.01 & \cellcolor[HTML]{EFEFEF}0.45 & 0.34                         & 0.34                         & 0.16    & 0.45    & 18/19 \\
% Spanish         & 0.58  & 0.7                          & 0.66                         & \cellcolor[HTML]{EFEFEF}*    & *       & 0.62    & 8/17  \\
% Telugu          & 0.44  & 0.67                         & \cellcolor[HTML]{EFEFEF}0.78 & 0.36                         & 0.29    & 0.78    & 16/24
% \end{tabular}
% \caption{Task scores for different methods}
% \label{tab:scores}
% \end{table*}
% Please add the following required packages to your document preamble:
% \usepackage{graphicx}
% \usepackage[table,xcdraw]{xcolor}
% Beamer presentation requires \usepackage{colortbl} instead of \usepackage[table,xcdraw]{xcolor}
\begin{table*}[htbp!]
%\resizebox{\textwidth}{!}{%
\small
\centering
\begin{tabular}{l|lllll|ll}
\rowcolor[HTML]{C0C0C0} 
 & \multicolumn{5}{l|}{\cellcolor[HTML]{C0C0C0}Train   Data} & \multicolumn{2}{l|}{\cellcolor[HTML]{C0C0C0}Dev Data} \\
LANGUAGE & \multicolumn{3}{l}{English Translation} & \multicolumn{2}{l|}{Multilingual Model} & Official score & Ranking \\
 & \begin{tabular}[c]{@{}l@{}}Spacy \\ Similarity\end{tabular} & cos vector & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}fine-tuned \\ SBERT\end{tabular}} & DBMCv1\footnote{distiluse-base-multilingual-cased-v1 \citep{sarzynska2021detecting}} & \begin{tabular}[c]{@{}l@{}}all-mpnet-\\ base-v2\footnotemark\end{tabular} & \multicolumn{2}{l}{} &  \\ \cline{1-6}
Algerian Arabic & 0.25 & 0.44 & \multicolumn{1}{l|}{\cellcolor[HTML]{EFEFEF}0.51} & 0.42 & 0.39 & 0.37 & 18/20 \\
Amharic & 0.37 & 0.61 & \multicolumn{1}{l|}{\cellcolor[HTML]{EFEFEF}0.78} & 0.16 & 0.12 & 0.78 & 11/16 \\
\textit{English} & \textit{0.34} & \textit{0.61} & \multicolumn{1}{l|}{\textit{\cellcolor[HTML]{EFEFEF}0.80}} & \textit{*} & \textit{*} & \textit{0.81} & \textit{10/34} \\
Hausa & 0.07 & 0.43 & \multicolumn{1}{l|}{\cellcolor[HTML]{EFEFEF}0.65} & 0.21 & 0.34 & 0.62 & 12/19 \\
Kinyarwanda & 0.18 & 0.39 & \multicolumn{1}{l|}{\cellcolor[HTML]{EFEFEF}0.57} & 0.3 & 0.38 & 0.57 & 8/14 \\
Marathi & 0.45 & 0.68 & \multicolumn{1}{l|}{\cellcolor[HTML]{EFEFEF}0.81} & * & * & 0.86 & 13/25 \\
Moroccan Arabic & -0.01 & \cellcolor[HTML]{EFEFEF}0.45 & \multicolumn{1}{l|}{0.34} & 0.34 & 0.16 & 0.45 & 18/19 \\
Spanish & 0.58 & 0.7 & \multicolumn{1}{l|}{0.66} & \cellcolor[HTML]{EFEFEF}* & * & 0.62 & 8/17 \\
Telugu & 0.44 & 0.67 & \multicolumn{1}{l|}{\cellcolor[HTML]{EFEFEF}0.78} & 0.36 & 0.29 & 0.78 & 16/24
\end{tabular}%
%}
\caption{Task scores for different methods}
\label{tab:scores}
\end{table*}
\footnotetext{{\url{https://huggingface.co/sentence-transformers/all-mpnet-base-v2} and \citet{reimers-2020-multilingual-sentence-bert}}}
\section{Experimental setup}
At the beginning stage of the experiment, we undertook an examination of several readily implementable models on the English baseline dataset and compared the predicted scores with human-labeled scores through Pearson correlation scores. 

In the initial English baseline model, we included the SpaCy similarity model\footnote{\url{https://spacy.io/usage/linguistic-features\#vectors-similarity}}, cosine vector similarity, and fine-tuned-BERT models\footnote{\url{https://github.com/AndriyMulyar/semantic-text-similarity}}. For the SpaCy similarity, we directly applied it to the training dataset, yielding a result of 0.34 (Pearson). In the case of cosine similarity, we tried out two methods of word embedding: 
\begin{enumerate}
    \item \textbf{Binary occurrence vectors}:
        This approach involves creating set-based word vectors using binary occurrence, combining them into a joint space, and comparing them using cosine similarity to quantify the relatedness between the original sets in vectorized forms. 
    \item \textbf{TF-IDF transformer-based vectors}:
        Using the TF-IDF vectorizer from the sklearn \citep{scikit-learn} library, we obtained TF-IDF weights for each word. The TF-IDF weight is proportional to the word's frequency in the document but is offset by its frequency in the corpus.    
\end{enumerate}
Upon comparing these two word-embedding methods, the Pearson correlation results did not reveal a significant difference. Therefore, we selected the Binary occurrence method as the cosine vector similarity, which achieved a score of  0.61 as indicated in Table 1. We use Pearson as opposed to Spearman rank correlation simply because that is what the original task description uses \citep{semrel2024dataset,ousidhoum2024semrel2024}.

The final component of the English baseline is the application of the fine-tuned BERT model to compute semantic relatedness with the (unfine-tuned) ClinicalBertSimilarity\footnotemark[5] and WebBertSimilarity\footnotemark[5] models and a batch size of 10 for both. The creators of the model claim that the ``project contains an interface to fine-tuned, BERT-based semantic text similarity models. It modifies pytorch-transformers by abstracting away all the research benchmarking code for ease of real-world applicability"\footnotemark[5]. This proved to be the most successful approach with a result of 0.8 for English. Although the task in question is about semantic relatedness, since many of the datasets involved in the creation of the datasets come from similarity data. Additionally, as similarity can be considered a subtype of relatedness, the use of similarity models seemed logical due to their wider availability compared to relatedness models. % Do I need to cite the model here?
%from semantic_text_similarity.models import WebBertSimilarity
%from semantic_text_similarity.models import ClinicalBertSimilarity

After establishing the English baseline, we evaluated several multilingual and language-specific BERT-based similarity models to assess textual relatedness (or similarity) across other language training datasets including the SBERT model for Telugu \citep{joshi2022l3cubemahasbert}, Sentence-BERT \cite{reimers-2019-sentence-bert}, BioLORD-2023\cite {remy-et-al-2023-biolord}, etc. However, the results were suboptimal, which is surprising since previous work has shown that sentence transformers show significant improvements to semantic similarity tasks, particularly cross-lingual tasks \citep{hammerl2023exploring}. Given the significantly better performance of the English baseline, we decided to translate all language datasets into English before applying the relatedness prediction models. In the case of Spanish we found that using distiluse-base-multilingual-cased-v1\citep{reimers-2019-sentence-bert} produced higher accuracies than the translation approach, and thus Spanish is the only language we did not translate to English.

When introducing the translation tools, we explored two approaches: utilizing a translation model (Machine Translation) and implementing Google Translate. 
\begin{enumerate}
    \item \textbf{Machine Translation}:    
    In the Machine Translation method, we applied M2M100 \cite{fan2020englishcentric} as the translation model. The model can directly translate between the 9,900 directions of 100 languages. 
    \item \textbf{Google Translate}:    
    For the machine translations, we utilized the deep-translator library\footnotemark, a versatile tool that facilitates simple language translation using multiple translators.
\end{enumerate}
\footnotetext{\url{https://deep-translator.readthedocs.io/en/latest/README.html\#id1}}
Despite the relatively high performance claimed by the M2M100 model as described by \citet{fan2020englishcentric}, the results after the translation process are less than 0.5 for all languages except Spanish, where it achieved a result of 0.67. In contrast, the Google Translate API demonstrated better performance during the training process with the English baseline model (detailed results are listed in Table \ref{tab:scores}).

Our multilayered approach mirrors that of \citet{jeyaraj2021mnet} although ours is a much simpler setup.


\section{Results}
%What were our findings

%Main quantitative findings: How well did your system perform at the task according to official metrics? How does it rank in the competition?
Our rankings show that our approach is nowhere near the state-of-the-art, but it is still a reliable option when more language-specific approaches are unavailable as is often the case with moderately low-resource languages. Out team ranked in the middle of the pack for most languages, but in the top third for English, Marathi, and Spanish, and the bottom for both Arabic dialects, which was expected. The rankings, scores, and models used for each submission can be seen in table \ref{tab:scores}. We analyze the results in the conclusions section.

%Quantitative analysis: Ablations or other comparisons of different design decisions to better understand what works best. Indicate which data split is used for the analyses (e.g. in table captions). If you modify your system subsequent to the official submission, clearly indicate which results are from the modified system.

%Error analysis: Look at some of your system predictions to get a feel for the kinds of mistakes it makes. If appropriate to the task, consider including a confusion matrix or other analysis of error subtypes—you may need to manually tag a small sample for this.


% \begin{table*}[htbp]
% \begin{tabular}{cl|ll}
% \rowcolor[HTML]{C0C0C0} 
% \multicolumn{1}{l}{\cellcolor[HTML]{C0C0C0}Languge} & Model                                    & Dice score                & Pearson                   \\ \hline
%                                                     & baseline                                 & 2                         & 3                         \\
%                                                     & \cellcolor[HTML]{EFEFEF}SpaCy similarity & \cellcolor[HTML]{EFEFEF}5 & \cellcolor[HTML]{EFEFEF}6 \\
% \multirow{-3}{*}{en}                                & BERT Sentence transformers               & 4                         & 4                         \\ \hline
%                                                     & baseline                                 & 2                         & 3                         \\
%                                                     & \cellcolor[HTML]{EFEFEF}SpaCy similarity & \cellcolor[HTML]{EFEFEF}5 & \cellcolor[HTML]{EFEFEF}6 \\
% \multirow{-3}{*}{ar}                                & BERT Sentence transformers               & 4                         & 4                        
% \end{tabular}
% \caption{Model comparison}
% \label{tab:scores}
% \end{table*}

% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% \usepackage[table,xcdraw]{xcolor}
% Beamer presentation requires \usepackage{colortbl} instead of \usepackage[table,xcdraw]{xcolor}








\section{Conclusions}
%What else could we have done
%What worked and didn't work
%We started with the simplest approaches such as SpaCy similarity measures to add to the original baseline measures...




%↓[Hidetsune]
To sum up, we first focused on English to have a good solution with fine-tuned BERT, and then we applied that solution to other languages by translating the sentences into English using machine translation. Since our English solution is reasonably good (rank 10/34, official score of .81), the application of the solution worked much better than using multilingual models in many languages including Amharic, Marathi, and Telugu for which there exist language-specific semantic similarity models. We speculate that the reason the \textit{MT+English model} worked better than the language-specific relatedness models is due to the higher quality and more diverse training data for the English model(s) as well as machine translation simplifying words to the most commonly used ones, artificially making similar sentences more similar. % might be due to the quite high precision of translation and the low performance of multilingual models, meaning that many multilingual models that claim high precision actually do not necessarily work as well as English models depending on the dataset.

The importance of an accurate machine translation can be seen in the failure of our approach with the Arabic dialects in particular. Google Translate does not have specific translators for Moroccan or Algerian Arabic, instead, we had to rely on general Arabic. This likely produced much lower quality translations obfuscating the semantic links between the sentence pairs making it difficult for the English model to accurately judge relatedness. This issue was further exacerbated by the fact that no one on our team speaks any of the languages in the task besides English, which made manual evaluations of the MT output difficult.

%On the other hand, the application of our English solutions did not work uniformly well with all languages, which is the main weakness of our approach. One of the main reasons is likely mistranslations when translating into English. For instance, the solution did not work well for Algerian Arabic and Moroccan Arabic, where we had to use general Arabic translation because Google translator does not deal with the difference between them.\par %
Darja and Darija are the names for Algerian and Moroccan Arabic respectively, and they are collectively known as Maghrebi Arabic. Due to its roots in Berber languages, there are notable distinctions between Maghrebi Arabic and Standard Arabic, and using the latter for these two dialects may yield a suboptimal result.

%Semantic similarity is usually described as a \textit{is a} relationship in computer science. Of course, this is known as hypernyms and hyponyms in linguistics. \textit{a car is a vehicle} (hyponym-hypernym) and \textit{bus} is similar to \textit{car} (co-hyponyms). An example from the task data would be \textit{Una mujer a punto de comer pescado} and \textit{Una mujer a punto de comer trucha} (\textit{A woman about to eat fish/trout}) where human evaluations had given the pair a 1.0 relatedness score.

%Additionally, the human-judged scores on the official development data do not necessarily reflect differences in word level strictly when it comes to hypernyms/hyponyms. For instance, \textit{Una mujer a punto de comer pescado} and \textit{Una mujer a punto de comer trucha} were translated as \textit{A woman about to eat fish} and \textit{A woman about to eat trout} respectively by MT and this pair had been given a 1.00 correlation by the human evaluation. %Our manual translations for these sentences are \textit{A woman is about to eat fish} and \textit{A woman is about to eat trout} respectively. Not only did MT miss a grammatically important word ("is"), 
%The hyponymic relation between "fish" and "trout" was not reflected in the human judgment, of course, if someone eats a trout, they are eating a fish, but if someone is eating a fish, they are not necessarily eating trout. It seems that our solutions did recognize this difference (scoring 0.67 by fine-tuned Bert for example), so this issue of strictness for semantic similarity and hyponymy recognition might be one of the things that needs to be discussed and implemented in future work. 

% ↑[Hidetsune] 


Curiously, a similar issue occurred with Spanish. Spanish is much more closely related to English than the other languages in subtask A, and therefore we expected our approach to get a fairly high score similar to English, especially considering the current state of machine translation between English and Spanish. However, it seems that translation of Spanish into English affects the semantic relations of the original sentences, which might be one of the main reasons causing the very low scores and making us choose the multilingual model for Spanish rather than the machine-translated one.

We hypothesize that one of the reasons that Google Translate worked so well on the low-resource languages most dissimilar from English might be because smaller training datasets for MT would force the translation to use less context and instead increase the reliance on individual lexical items leading to sentence pairs with high relatedness becoming more similar via translation. For languages with better MT models, it is conceivable that the better translations work against this approach as it might make the sentence pairs less similar as reflected by the higher scores for Spanish using multilingual models, and the very low scores for both Arabic dialects. In future work, it might be worthwhile to use mixed methods starting with language-specific models and then expanding to incorporate machine translation and larger models developed for, e.g., English.

% Entries for the entire Anthology, followed by custom entries

\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}


\end{document}
