% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\usepackage{graphicx}

\usepackage{adjustbox}

\title{DeBERTa at SemEval-2024 Task 9: Using DeBERTa for Defying Common Sense}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Marco Siino \\
Department of Electrical, Electronic \\and
Computer Engineering \\
  University of Catania \\
  Italy \\
  \texttt{marco.siino@unipa.it} }

\begin{document}
\maketitle
\begin{abstract}



The significant achievements of language models have motivated researchers in the natural language processing (NLP) community to confront challenges requiring nuanced and implicit reasoning, inspired by human-like common-sense understanding. Although efforts focusing on vertical thinking tasks have received substantial recognition, there remains a notable lack of investigation into lateral thinking puzzles. To bridge this void, the authors at SemEval-2024 propose BRAINTEASER: a multiple-choice Question Answering task designed meticulously to assess the model's lateral thinking capabilities and its capacity to question default common-sense assumptions. Specifically, at the SemEval-2024 Task 9, for the first subtask (i.e., Sentence Puzzle) the organizers asked the participants to develop models able to reply to multi-answer brain-teasing questions. For this purpose, we propose the application of a DeBERTa model in a zero-shot configuration. The proposed approach achieves an aggregate score of 0.250. Suggesting a significant room for improvements in future works.

\end{abstract}

\section{Introduction}

Human reasoning encompasses two fundamental types of cognitive processing: vertical and lateral thinking. Vertical thinking is marked by its sequential and analytical approach, drawing upon principles of rationality, logic, and rule-following, often attributed to the left-brain hemisphere \cite{knauff2013space,huang2023lateval}. This mode of thinking is essential for creating logical pathways, such as understanding physical scenarios or solving riddles based on direct associations. In contrast, lateral thinking, often referred to as "thinking outside the box," is a creative cognitive process. It entails exploring problems from unconventional perspectives and challenging ingrained assumptions. Lateral thinking, associated with the right-brain hemisphere, is crucial for resolving unconventional puzzles by defying common-sense associations and considering alternative perspectives.

While natural language processing (NLP) models have made significant strides in vertical thinking tasks, particularly in the field of large language models (LLMs). Their performance in lateral thinking remains largely unexplored. LLMs have demonstrated remarkable performance across various reasoning tasks, even when provided with minimal or no training examples. These models excel in tasks requiring vertical thinking abilities, such as reasoning over physical interactions and social implications \cite{siino2022mcrock}, showcasing strong common-sense association and inference capabilities. However, prior research has largely overlooked the evaluation of LLMs' lateral thinking abilities, as creative thinking problems are often filtered out during data preprocessing, and only those aligned with common-sense associations are retained.

To address this gap, a novel benchmark called BRAINTEASER \cite{jiang-etal-2023-brainteaser} to evaluate the lateral thinking abilities of state-of-the-art LLMs is proposed at SemEval-2024 Task 9 \cite{jiang-semeval-2024-brainteaser}. The organizers frame lateral thinking puzzles as multiple-choice Question Answering (QA) tasks, a format that is intuitive for humans to engage with and straightforward to assess automatically. The BRAINTEASER benchmark comprises two tasks: Sentence Puzzles and Word Puzzles, designed to assess lateral thinking at different levels of granularity. To develop the dataset, organizers employ a data collection pipeline that retrieves relevant puzzles from publicly available websites, filters out irrelevant question categories, and ensures high data quality. Additionally, to mitigate concerns regarding LLM memorization and consistency, the organizers enhance BRAINTEASER with two reconstruction strategies: semantic reconstruction and context reconstruction. These strategies aim to promote deeper understanding and reasoning rather than mere memorization of patterns.



To meet these objectives, there is a growing demand for automated tools capable of understanding data using recent advancements in NLP models. The emergence of machine and deep learning architectures has sparked increased interest in NLP, prompting substantial efforts to develop techniques for automated identification and understanding of textual content available on the internet. In the literature, various strategies have been proposed so far. Over the past fifteen years, some of the most successful approaches have included Support Vector Machines (SVM) \cite{colas2006comparison,croce2022svm}, Convolutional Neural Networks (CNN) \cite{kim2014convolutional,siino2021detection}, Graph Neural Networks (GNN) \cite{lomonaco2022courage}, ensemble models \cite{miri2022ensemble,siino2022t100}, and Transformers \cite{vaswani2017attention}. 

The sections of this paper are structured as follows: Section \ref{sec:background} offers background information on Task 9, held at SemEval-2024. In Section \ref{sec:system_overview}, we outline the approach introduced in this study. Section \ref{sec:experimental_setup} delves into the specifics of the experimental setup employed to reproduce our findings. The outcomes of the official task and relevant discussions are presented in Section \ref{sec:results}. Finally, Section \ref{sec:conclusion} concludes our study and suggests avenues for future research.

We make all the code publicly available and reusable on GitHub\footnote{\raggedright\url{https://github.com/marco-siino/SemEval2024/}}.

\section{Background}
\label{sec:background}

The increasing adoption of Transformer-based architectures in academic research has also been bolstered by various methodologies showcased at SemEval 2024. These methodologies tackle diverse tasks and yield noteworthy findings. For instance, at the Task 2 \citep{jullien-etal-2024-semeval}, where to address the challenge of identifying the inference relation between a plain language statement and Clinical Trial Reports is used T5 \cite{siino2024t5medical}; Task 4 \citep{dimitrov2024semeval} where is employed a Mistral 7B model to detect persuasion techniques in memes \cite{siino2024mcrock}; and Task 8 \citep{wang2024semeval}, that utilizes a DistilBERT model to identify machine-generated text \cite{siino2024badrock}.

The Task 9 hosted at SemEval-2024, is based on the human reasoning processes comprising the two already-mentioned types of thinking: vertical and lateral. 

Specifically, the BRAINTEASER QA task consists of two subtasks: the Sentence and the Word Puzzle ones, that require awareness of common-sense “defaults” and overwriting them through unconventional thinking that distinguishes these defaults from hard constraints.

In detail, for the Sentence Puzzle one, the puzzle defying common-sense is centred on sentence snippets. On the other hand, for the Word Puzzle subtask, the response diverges from the conventional interpretation of the word and concentrates on the letter arrangement within the target question.



Both subtasks incorporate an adversarial subset, crafted by manually altering the original brain-teasers while preserving their underlying reasoning paths.

An example from the official CodaLab page\footnote{\url{https://codalab.lisn.upsaclay.fr/competitions/15566}} takes as example the following original sentence:

\begin{quote}
\textit{"A man shaves everyday, yet keeps his beard long."}

\end{quote}

The four possible explanations are:


\begin{enumerate}
    \item \textbf{He is a barber.}
    \item He wants to maintain his appearance.
    \item He wants his girlfriend to buy him a razor.
    \item None of the above.
\end{enumerate}

However, the task organizers also included two other samples based on  the previous one. In these two cases, a semantic and a contextual reconstruction have been made to challenge a classification model. The two reconstructions (with the same four possible explanations as in the original) are:

\begin{itemize}
    \item SEMANTIC RECONSTRUCTION: \textit{"A man preserves a lengthy beard despite shaving every day."}
    \item CONTEXT RECONSTRUCTION: \textit{"Tom attends class every day but doesn’t do any homework."}
\end{itemize}




\section{System Overview}
\label{sec:system_overview}


Despite evidence suggesting that Transformers may not always yield optimal results for every text classification task \cite{siino2022fake}, various strategies, such as domain-specific fine-tuning \cite{sun2019fine, van2023vietnamese} and data augmentation \cite{lomonaco2023text, mangione2022improving,siino2024backtranslate}, have proven to be advantageous depending on the task's objectives. 

However, to address the task 9 hosted at SemEval-2024 we made use of a zero-shot learning approach \cite{Chen202312844,Wahidur202410146}, making use of the DeBERTa Transformer \cite{he2020deberta}. 


Our approach is zero-shot \cite{pourpanah2022review} and make use of the above-mentioned DeBERTa model. Specifically, we employed the multilingual version 3 fine-tuned on the SQuAD2.0 dataset\footnote{\url{https://rajpurkar.github.io/SQuAD-explorer/}}. DeBERTa improves upon the BERT and RoBERTa models by introducing disentangled attention mechanisms and an enhanced mask decoder. Leveraging these enhancements, DeBERTa outperforms RoBERTa across most Natural Language Understanding (NLU) tasks when trained on a dataset of 80GB in size. In DeBERTa V3, efficiency is further enhanced by integrating ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Comparative analysis against DeBERTa reveals notable enhancements in model performance across downstream tasks in the V3 version. Further elaboration on the novel techniques employed in this new model can be found in the original paper. The version of DeBERTa utilized in our experiments is mDeBERTa, a multilingual variant of DeBERTa. It maintains an identical architecture while being trained on CC100 multilingual data. The mDeBERTa V3 base model comprises 12 layers with a hidden size of 768. It encompasses 86 million backbone parameters and a vocabulary of 250,000 tokens, resulting in 190 million parameters in the embedding layer. This model underwent training using 2.5 trillion tokens of CC100 data, akin to the XLM-R model.

For the experimental settings, we started evaluating several prompt engineering strategies \cite{white2023prompt,liu2023jailbreaking} to optimize the model replies and to obtain satisfactory results guided by the labelled samples in the training set. For example, we included in the prompt/question to the model, the premise that the given question is a brain-teaser one. Furthermore, we also evaluated the performance of the model on the training set using a few-shot learning setup. In this case, we provided as input (included in the prompt) ten questions indicating the correct answer. Also in this case we did not obtain satisfactory results. 

More specifically, given the task hosted at SemEval-2024, we asked the model: \textit{"What is the correct answer to the brain teaser question from the following choices? (Pick only one Option (A)-(D)"}. To this request, the model replied with one or more words that we parsed to extract one of the choices. For example, given the context: 

\begin{quote}
    \textit{"Romeo and Juliet are discovered dead on the bedroom floor. Glass shards and some water were on the floor when they were found. A bookcase and a bed are the sole pieces of furniture in the space. Other than the neighboring railroad track, the house is located in a rural area. How is that even doable?  "} 
\end{quote}


And our question:

\begin{quote}
    \textit{"What is the correct answer to the brain teaser question from the following choices? (Pick only one Option (A)-(D)"} 
\end{quote}

And the answers/options: 

\begin{quote}
  (A): They were sleeping and scared by the sound of track.

  (B): The rumble of the train moved the shelf which 
crushed them.

  (C): Romeo and Juliet are fish. The rumble of the train knocked the tank off the shelf, it broke and Romeo and Juliet did not survive.

  (D): None of above.
\end{quote}




The model replied with: 

\begin{quote}
\textit{"Romeo and Juliet are fish."} 
\end{quote}

that we mapped into the label \textit{2} corresponding to the third answer. Finally, we collected all the predictions provided on the test set to into a JSON file with required format to submit our predictions.

During our experiments to build our prompt, we also evaluated other LLMs like GPT-Neo and GPT-NeoX \cite{gao2020pile}. However, on the  labelled training set, we found better performance of DeBERTa in the responses provided. It is also worth notice that we conducted several experiments to find an effective prompt strategy to address the task.



As indicated in a recent investigation by Siino et al. \cite{siino2024text}, preprocessing does not significantly impact text classification tasks when employing Transformers. Specifically, the optimal combination of preprocessing strategies closely resembles the performance achieved without any preprocessing at all, particularly in the context of Transformer models. Therefore, to maintain a highly efficient and computationally lightweight system, we opted not to apply any preprocessing to the text.


\section{Experimental Setup}
\label{sec:experimental_setup}

We implemented our model on Google Colab. The library we used come from Hugging Face and as already mentioned is a multilingual version of DeBERTa\footnote{\url{https://huggingface.co/timpal0l/mdeberta-v3-base-squad2}}. The dataset provided for all the phases are available on the official competition page. We did not perform any additional fine-tuning on the model. To run the experiment, a T4 GPU from Google has been used. After the generation of the predictions, we exported the results on the format required by the organizers. As already mentioned, all of our code is available on GitHub.


\section{Results}
\label{sec:results}

Participants in Brain Teaser may participate in any or all of the two subtasks. The organizers created two adversarial questions, semantic and context reconstruction, for each brain-teaser (examples can be found on the Task Home Page). The evaluation metrics applied are Instance-based Accuracy and Group-based Accuracy, defined as follows:

\begin{itemize}
    \item Instance-based Accuracy: Each question, whether original or adversarial, is treated as an individual instance. Accuracy is reported for the original question, its semantic reconstruction, and context reconstruction.
    \item Group-based Accuracy: Questions and their corresponding adversarial instances are grouped together. A system earns a score of 1 only if it correctly answers all questions within the group. Accuracy is reported for original and semantic reconstruction and original and semantic and context reconstruction.
\end{itemize}





In Table \ref{tab:deberta_results}, we present the outcomes derived from our methodology. They are the same results publicly available on the official final ranking shown on the official task page\footnote{\url{https://codalab.lisn.upsaclay.fr/competitions/15566}}. The results are about the sentence task, given the fact that we did not take part in the word-related task.

Table \ref{tab:competition_results} presents the performance results of the top three teams alongside the results achieved by the final-ranking team, as displayed on the official task page. While our straightforward approach shows potential for enhancement compared to the top-performing models, it is noteworthy that our method required no additional pre-training. Moreover, the computational resources needed to address the task were manageable, utilizing the free online resources provided by Google Colab.


\begin{center}
\begin{table}
\begin{center}
\begin{tabular}{ cc } 
    \hline
    \multicolumn{2}{c}{DeBERTa} \\
      \hline
  Original & 0.225 \\
  Semantic & 0.250 \\
  Context & 0.275 \\
  Ori+Sem & 0.200 \\
  Ori+Sem+Con & 0.075 \\
  Overall &  0.250\\ 
  \hline

\end{tabular}
\end{center}
\caption{\label{tab:deberta_results}The method's performance on the test set. In the table are reported the results obtained and shown on the official task page. }
\end{table}
\end{center}



\begin{center}
\begin{table*}[ht]
\begin{center}
\begin{tabular}{ ccccccc } 
    \hline
   TEAM NAME & Original & Semantic & Context & Ori+Sem & Ori+Sem+Con & Overall \\ 
  \hline
  	
abdelhak (1) & 1.000 & 1.000 & 0.950 & 1.000 & 0.950 & 0.983 \\ 
  \hline
  lulu13gjdfnglgr (2) & 1.000 & 0.975 & 0.925 & 0.975 & 0.900 & 0.967 \\ 
   \hline
  Maxine (3) & 0.975 & 0.975 & 0.925 & 0.975 & 0.900 & 0.958 \\
  \hline
  wwangbw (31) & 0.300 &0.175 & 0.150 & 0.075 & 0.025 & 0.208 \\ 
  \hline
\end{tabular}
\end{center}
\caption{\label{tab:competition_results}Comparing performance on the test set. In the table are shown the results obtained by the first three teams and by the last one.  In parentheses is reported the position in the official final ranking.}
\end{table*}
\end{center}


\section{Conclusion}
\label{sec:conclusion}


This paper introduces the utilization of a DeBERTa model for addressing Task 9 at SemEval-2024. In our submission, we opted for a zero-shot learning approach, leveraging a pre-trained and fine-tuned Transformer model without further adaptation. Through various experiments, we found it advantageous to construct a prompt containing the question for the model. Subsequently, we provided the context, question, and answer candidates as the prompt, prompting the model to discern the correct candidate answer. Despite the task's inherent complexity, as evidenced by the final ranking, there remains ample room for improvement.

Potential alternative methodologies include leveraging the few-shot learning capabilities of the model or exploring alternative models such as GPT and T5. Additionally, integrating additional data or incorporating samples from training and development sets could yield performance enhancements. Further refinements could be achieved through fine-tuning and framing the problem as a text classification task. Moreover, given the promising results observed across various tasks, the adoption of few-shot learning or data augmentation strategies could also be explored for improved outcomes \cite{wang2023few, maia2024transformers, siino2023profiling, meng2024hierarchical, muftie2023indobert, tapia2020data, siino2023xlnet}.

While our straightforward approach demonstrates potential for refinement, it is noteworthy that it required no additional pre-training. Moreover, the computational resources needed to address the task were manageable, utilizing the free online resources provided by Google Colab.


\section*{Acknowledgments}
We extend our gratitude to the anonymous reviewers for their insightful comments and valuable suggestions, which have significantly enhanced the clarity and presentation of this paper.

% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}

%\appendix

%\section{Appendix}
%\label{sec:appendix}




\end{document}
