% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{tikz}
\usetikzlibrary{shapes,arrows,positioning}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\usepackage{graphicx}

\usepackage{adjustbox}

\title{TransMistral at SemEval-2024 Task 10: Using Mistral 7B for Emotion Discovery and Reasoning its Flip in Conversation}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Marco Siino \\
Department of Electrical, Electronic \\and
Computer Engineering \\
  University of Catania \\
  Italy \\
  \texttt{marco.siino@unipa.it} }

\begin{document}
\maketitle
\begin{abstract}

The EDiReF shared task at SemEval 2024 comprises three subtasks: Emotion Recognition in Conversation (ERC) in Hindi-English code-mixed conversations, Emotion Flip Reasoning (EFR) in Hindi-English code-mixed conversations, and EFR in English conversations. The objectives for the ERC and EFR tasks are defined as follows: 1) Emotion Recognition in Conversation (ERC): In this task, participants are tasked with assigning an emotion to each utterance within a dialogue from a predefined set of possible emotions. The goal is to accurately recognize and label the emotions expressed in the conversation; 2) Emotion Flip Reasoning (EFR): This task involves identifying the trigger utterance(s) for an emotion-flip within a multi-party conversation dialogue. Participants are required to pinpoint the specific utterance(s) that serve as catalysts for a change in emotion during the conversation. In this paper we only address the first subtask (ERC) making use of an online translation strategy followed by the application of a Mistral 7B model together with a few-shot prompt strategy. Our approach obtains an F1 of 0.36, eventually exhibiting further room for improvements.

\end{abstract}

\section{Introduction}


Affective computing has experienced a resurgence, largely propelled by recent advancements in artificial intelligence. Emotion Recognition in Conversations (ERC) has emerged as a prominent task within affective computing, garnering increasing attention \cite{poria2019emotion,kumar2023multilingual}. Its objective is to discern the emotion conveyed in each utterance during conversations, with implications for various applications including the development of effective dialogue systems, facilitating social viewpoint mining, and creating intelligent medical systems. Current research in ERC primarily focuses on capturing the emotional state of speakers through contextual analysis and establishing distinct contexts for different speakers, often leveraging multimodal data to support this endeavour. Despite recent strides, two major challenges persist: (1) Ensuring emotional consistency and (2) Generating contextual information. Current research efforts broadly fall into two categories: the first involves obtaining contextual representations of utterances using temporal neural networks, while the second entails capturing long-distance information through graph networks. However, these approaches overlook a crucial aspect: changes in utterance order can alter the meaning of utterances, potentially leading to varying emotional expressions. A shift in utterance order impacts the underlying meaning of the utterance, consequently influencing the speakers' emotions. 

The increasing demand for automated tools capable of extracting and categorizing data from online sources underscores the need to address both established and emerging societal concerns efficiently. Recent strides in machine and deep learning architectures have sparked significant interest in Natural Language Processing (NLP). Efforts have been intensified towards developing techniques for automating the identification and categorization of textual content prevalent on the internet today. In the literature, various strategies have been proposed for performing text classification tasks. Over the past fifteen years, some of the most successful strategies have included Support Vector Machines (SVM) \cite{colas2006comparison, croce2022svm}, Convolutional Neural Networks (CNN) \cite{kim2014convolutional, siino2021detection}, Graph Neural Networks (GNN) \cite{lomonaco2022courage}, ensemble models \cite{miri2022ensemble, siino2022t100}, and more recently, Transformers \cite{vaswani2017attention, siino2022mcrock}.

At SemEval-2024 Task 10 \cite{kumar2024semeval} – Emotion Discovery and Reasoning its Flip in Conversation (EDiReF) – three Subtasks were proposed. All the three subtasks are presented and discussed in the Section \ref{sec:background}.

To face with the first subtask (ERC), we proposed a Transformer-based approach which made use of Mistral 7B \cite{jiang2023mistral}. We used the model in a particular few-shot way described in the rest of this paper. Specifically, after translating all the code-mixed samples in English, we provided the samples from the labelled train and dev set to the model, asking while prompting to predict the emotion for the current utterance (i.e., the current sample from the test set).  

The rest of the paper is made as follows. In Section \ref{sec:background} we provide some background on the Task 10 hosted at SemEval-2024. In Section \ref{sec:system_overview} we provide a description of the approach presented. In Section \ref{sec:experimental_setup} we provide details about the experimental setup to replicate our work. In Section \ref{sec:results}, the results of the official task and some discussions are provided. In section \ref{sec:conclusion} we present our conclusion and proposals for future works.


We make all the code publicly available and reusable on GitHub\footnote{\raggedright\url{https://github.com/marco-siino/SemEval2024/}}.

\section{Background}
\label{sec:background}


This section furnishes background information regarding Task 10 (Subtask 1), held at SemEval-2024. 


The EDiReF shared task at SemEval 2024 \citep{kumar2024semeval} encompasses three distinct subtasks, namely: Emotion Recognition in Conversation (ERC) within Hindi-English code-mixed conversations, Emotion Flip Reasoning (EFR) within Hindi-English code-mixed conversations, and EFR within English conversations \citep{kumar2022discovering,kumar2023emotion}. The ERC task involves the assignment of emotions to each utterance within a dialogue, drawn from a predefined set of potential emotions. Conversely, the EFR task focuses on identifying the trigger utterance(s) responsible for inducing an emotion-flip within a multi-party conversation dialogue.


In the Figure \ref{fig:sample}, is reported a sample from the official competition website\footnote{\url{https://lcs2.in/SemEval2024-EDiReF/}} and specifically related to the Subtask 1 (i.e., ERC).

\begin{figure}
    \centering
    \includegraphics[width=1.0\linewidth]{samples.png}
    \caption{Example of some samples from the dataset. In this case, we report samples related to the first ERC task in which we took part. }
    \label{fig:sample}
\end{figure}


For Subtask 1, a submission entails a singular JSON file with each emotion in a new line. Every emotion correspond to each utterance in the official provided test set. 


The second and the third subtasks differ on the language used. In one case, the language is code-mixed Hindi-English and in another case only in English. In both cases, the participants were asked to propose automatic detection systems able to detect a trigger utterance that determined a changed in the emotion. Also in this case, an example from the official task webpage is reported in the Figure \ref{fig:sample_2}. The EFR sample contains a trigger (i.e., 1) in proximity of the fourth sentence contained in the dialogue. 

\begin{figure}
    \centering
    \includegraphics[width=1.0\linewidth]{sample2.png}
    \caption{Example of some samples from the dataset. In this case, we report samples related to the EFR task. }
    \label{fig:sample_2}
\end{figure}



\section{System Overview}
\label{sec:system_overview}

\begin{figure*}[!ht]
    \centering
    \resizebox{\linewidth}{!}{%
\begin{tikzpicture}[node distance = 2cm, auto]
    % Define styles
    \tikzstyle{block} = [rectangle, draw, text width=3cm, text centered, rounded corners, minimum height=1cm]
    \tikzstyle{line} = [draw, -latex']
    
    % Place nodes
    \node [block] (train_dev) {Train+Dev Sets};
    \node [block, below=0.1cm of train_dev] (test) {Test Set};
    \node [block, right= 2.7cm of train_dev] (deep_translator) {Deep Translator};
    \node [block, below=0.1cm of deep_translator] (deep_translator2) {Deep Translator};
    \node [block, minimum height=1.8cm, below right=-0.9cm and 2.7cm of deep_translator, text width=2cm, align=center] (mistral) {Mistral 7B};
    
    % Draw arrows
    \draw [line] (train_dev) ++(2.0,0.0) -- +(2,0) node[midway,above] {CMS Few-Shot};
    \draw [line] (test) ++(2.0,0.0) -- +(2,0) node[midway,below] {"hailo varld"};
    \draw [line] (deep_translator) ++(2,0.0) -- +(2,0) node[midway,above] {EN Few-Shot};
    \draw [line] (deep_translator2) ++(2,-0) -- +(2,0) node[midway,below] {"Hello World"};
    \draw [line] (mistral)  ++(1.3,-0.) -- +(1,0) node[midway,above] {Joy}; %++(2.2,0);
\end{tikzpicture}
}
 \caption{The system overview of our proposed approach. Given a set of Code Mixed Samples (i.e., CMS) from the train and dev sets, they are translated to English using Deep Translator (i.e., ENglish samples). Then they are all provided as input - a few-shot samples from the training set, together with the emotion definitions - to Mistral 7B. Following these few shot samples and the definitions as input, there is one utterance from the test set for which the emotion has to be predicted.}
    \label{fig:blockdiagram}
\end{figure*}


While it has been established that Transformers may not always be the optimal choice for text classification tasks \cite{siino2022fake}, the efficacy of various strategies, such as domain-specific fine-tuning \cite{sun2019fine, van2023vietnamese} and data augmentation \cite{lomonaco2023text, mangione2022improving}, depends on the specific objectives.

The increasing adoption of Transformer-based architectures in academic research has also been bolstered by various methodologies showcased at SemEval 2024. These methodologies tackle diverse tasks and yield noteworthy findings. For instance, at the Task 2 \citep{jullien-etal-2024-semeval}, where to address the challenge of identifying the inference relation between a plain language statement and Clinical Trial Reports is used T5 \cite{siino2024t5medical}; Task 4 \citep{dimitrov2024semeval} where is employed a Mistral 7B model to detect persuasion techniques in memes \cite{siino2024mcrock}; and Task 8 \citep{wang2024semeval}, that utilizes a DistilBERT model to identify machine-generated text \cite{siino2024badrock}.

Our approach is few-shot \cite{littenberg2022comparing} and make use of Mistral 7B. Mistral 7B, a language model boasting 7 billion parameters, is engineered to excel in both performance and efficiency. In comparison to the leading open 13B model (Llama 2), Mistral 7B demonstrates superior performance across all assessed benchmarks. Furthermore, it surpasses the top released 34B model (Llama 1) in tasks related to reasoning, mathematics, and code generation. The model capitalizes on grouped-query attention (GQA) to expedite inference, complemented by sliding window attention (SWA) to effectively process sequences of varying lengths while minimizing inference costs. Additionally, a fine-tuned variant, Mistral 7B – Instruct, is tailored for adhering to instructions, and it outperforms Llama 2 13B – chat model across both human and automated benchmarks. The introduction of Mistral 7B Instruct underscores the ease with which the base model can be fine-tuned to achieve notable performance enhancements. 

For our task, before prompting the model with the current sample from the test set, we made an online and real-time use of \textit{Google Translator} from the \textit{deep\_translator}\footnote{\url{https://pypi.org/project/deep-translator/}} library. Then we randomly selected eighty samples from the provided labelled training set and other eighty from the provided labelled dev set. Then we formatted the samples in each set in the following way:

\begin{quote}
\textit{speaker1 - utterance1 - emotion1 \\
speaker2 - utterance2 - emotion2 \\
...\\
speakerX - utterance80 - emotionY}
\end{quote}


After merging the formatted samples from both the training and the dev set, we fed the model, appending to the few-shot samples the current unlabelled sample from the official test set. At this point, the full text containing the few-shot samples plus the sample to be classified were provided as prompts to Mistral.


Then the question provided as prompt to the model was: \textit{" Use the CONTEXT to complete the SENTENCE using ONLY one emotion among: disgust, joy, neutral, anger, sadness, contempt, surprise, fear. Do not explain!"}. Where the \textit{CONTEXT} were the few-shot samples provided. For all the samples from the test set the model correctly predicted one of the available emotions from the list provided and shown in the Table \ref{tab:emotions}.


\begin{table}
    \centering
    \begin{tabular}{|c|}
    \hline
    EMOTIONS \\
    \hline
       1. \textit{disgust}\\
 2. \textit{joy}\\
 3. \textit{neutral}\\
 4. \textit{anger}\\
 5. \textit{sadness}\\
 6. \textit{contempt}\\
 7. \textit{surprise}\\
 8. \textit{fear} \\
       \hline
    \end{tabular}
    \caption{The list of all the motions available for the task. }
    \label{tab:emotions}
\end{table}

As noted in the recent study by \cite{siino2024text}, the contribution of preprocessing for text classification tasks is generally not impactful when using Transformers. More specifically, the best combination of preprocessing strategies is not very different from doing no preprocessing at all in the case of Transformers. For these reasons, and to keep our system fast and computationally light, we have not performed any preprocessing on the text.


\section{Experimental Setup}
\label{sec:experimental_setup}

Our model implementation was executed on Google Colab, utilizing the Mistral 7B library from Hugging Face, specifically the Mistral-7B-Instruct-v0.2-GGUF \footnote{\url{https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF}} version. Additionally, we utilized the \textit{deep\_translator} package with Google Translator \footnote{\url{https://pypi.org/project/deep-translator/}} for the translation task. The Mistral 7B version employed represents an enhanced iteration of the Mistral-7B-Instruct-v0.1 model, geared towards instruction fine-tuning. Instructions for instruction fine-tuning should be enclosed within [INST] and [/INST] tokens, with the initial instruction beginning with a sentence identifier, and subsequent instructions omitting this identifier. The generation process is terminated by the end-of-sentence token ID. Furthermore, we imported the Llama library \cite{touvron2023llama} from \textit{llama\_cpp}, with comprehensive details available on GitHub \footnote{\url{https://github.com/ggerganov/llama.cpp}}.

All datasets required for the various phases of the experiment are accessible on the Official Competition page. No additional fine-tuning was conducted on the model. The experiment was executed using a T4 GPU provided by Google. Upon generating the predictions, the results were exported in the format specified by the organizers. As previously mentioned, our complete codebase is accessible on GitHub.


\section{Results}
\label{sec:results}

As described on the official task page\footnote{\url{https://codalab.lisn.upsaclay.fr/competitions/16769}}, the evaluation criteria for the three tasks are delineated as follows:

\begin{itemize}
    \item Task 1 (ERC for code-mixed): Weighted F1 score computed across all emotions.
    \item Task 2 (EFR for code-mixed): F1 score computed specifically for triggers (label '1.0').
    \item Task 3 (EFR for English): F1 score computed for triggers (label '1.0') in English.
\end{itemize}

To generate the prediction file:

\begin{enumerate}
    \item For Task 1: Each line should depict an emotion associated with an utterance, with no additional lines separating dialogues. Each emotion should be in lowercase, devoid of extra spaces.
    \item For Tasks 2 and 3: Each line should represent either 0.0 or 1.0, reflecting the label of triggers assigned to an utterance. The formatting should adhere to a string format with a floating precision of 1 (e.g., 0.0 or 1.0, rather than 0 or 1).
\end{enumerate}

Then, it was asked to aggregate the outputs from all tasks into a single file named 'answer.txt', structured as follows:
\begin{itemize}
    \item Lines 1–1580: Predictions for Task 1.
    \item Lines 1581–9270: Predictions for Task 2.
    \item Lines 9271–17912: Predictions for Task 3.
\end{itemize}

Upon compilation, the organizers asked to create a zip file encompassing \textit{'answer.txt'} and proceed with submission as per guidelines.



\begin{center}
\begin{table}
\begin{center}
\begin{tabular}{ cccc } 
    \hline
  & T1-F1 & T2-F1 & T3-F1 \\ 
  \hline
  TransMistral 7B & 0.36 & 0.10 & 0.22 \\ 
  \hline
\end{tabular}
\end{center}
\caption{\label{tab:mistral_results}The method's performance on the test set. Even if we did not participate in the tasks 2 and 3, in the \textit{answer} file we included a list of 0s to complete all the lines as suggested by the task organizers.}
\end{table}
\end{center}

In the Table \ref{tab:mistral_results} we report the result obtained by the proposed approach on the official test set. Thanks to our application of an online translation followed by a Mistral 7B we have been able to reach the twenty-second position on the final ranking for the evaluation phase.



The Table \ref{tab:competition_results} presents the performance outcomes achieved by the top three teams and the last-ranked team, as delineated on the official task page. While our simplistic approach showcases potential for enhancement in comparison to the leading models, it is noteworthy that our method necessitated no additional pre-training. Moreover, the computational resources utilized to tackle the task remained feasible, courtesy of the complimentary resources provided by Google Colab.


\begin{center}
\begin{table*}[ht]
\begin{center}
\begin{tabular}{ cccc } 
    \hline
  TEAM NAME & T1-F1 & T2-F1 & T3-F1 \\ 
  \hline
  MasonTigers & 0.78 (1) & 0.79 (2) & 0.79 (1) \\ 
  \hline
  Knowdee & 0.73 (2) & 0.66 (4) & 0.61 (9) \\ 
   \hline
  IASBS & 0.70 (3) & 0.12 (7) & 0.25 (12) \\
  \hline
  GAVx & 0.08 (34) & 0.79 (2) & 0.76(2) \\ 
  \hline
\end{tabular}
\end{center}
\caption{\label{tab:competition_results}Comparing performance on the test set. In the table are shown the results obtained by the first three users and by the last one ordered considering the first task. In parentheses is reported the position for each task in the official final ranking.}
\end{table*}
\end{center}



\section{Conclusion}
\label{sec:conclusion}


This paper presents the application of Mistral 7B-model for addressing the Task 10 at SemEval-2024. For our submission, we decided to follow few-shot learning approach, employing as-is, an in-domain pre-trained Transformer. After several experiments, we found it beneficial to build a prompt containing some samples from the training and from the dev set. Then we provide as a prompt the current translated sample together with the few-shot samples. The model was asked to select one of the emotion among the ones available. The task presents inherent challenges, with evident scope for refinement, as underscored by the final ranking. Potential alternative methodologies encompass leveraging the zero-shot capabilities inherent in other models such as GPT and T5, expanding the training set size through the incorporation of additional data, or adopting alternative strategies for integrating ontology-based domain knowledge beyond the approaches delineated in our study. Furthermore, refinement opportunities exist through fine-tuning and recontextualizing the problem as a text classification task.

 Furthermore, given the interesting results recently provided on a plethora of tasks, also other few-shot learning \citep{wang2023few, maia2024transformers, siino2023profiling, meng2024hierarchical} or data augmentation strategies \citep{muftie2023indobert, siino2024backtranslate, tapia2020data, siino2023xlnet} could be employed to improve the results. Looking at the final ranking, our simple approach exhibits some room for improvements. However, it is worth notice that it required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by Google Colab. Also, thanks to the proposed approach, we have been able to outperform the baseline provided by the task organizers.

\section*{Acknowledgments}
We express our sincere appreciation to the anonymous reviewers for their constructive feedback and invaluable suggestions. Their insightful comments have greatly contributed to the refinement and clarity of this paper.

% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}

%\appendix

%\section{Example Appendix}
%\label{sec:appendix}

%This is an appendix.

\end{document}
