% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\usepackage{graphicx}

\usepackage{adjustbox}

\title{McRock at SemEval-2024 Task 4: Mistral 7B for Multilingual Detection of Persuasion Techniques in Memes}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Marco Siino \\
Department of Electrical, Electronic \\and
Computer Engineering \\
  University of Catania \\
  Italy \\
  \texttt{marco.siino@unipa.it} }

\begin{document}
\maketitle
\begin{abstract}

One of the most widely used content types in internet misinformation campaigns is memes. Since they can readily reach a big number of users on social media sites, they are most successful there. Memes used in a disinformation campaign include a variety of rhetorical and psychological strategies, including smearing, name-calling, and causal oversimplification, to achieve their goal of influencing users. The shared task's objective is to develop models for recognizing these strategies solely in a meme's textual content (Subtask 1) and in a multimodal context where both the textual and visual material must be analysed simultaneously (Subtasks two and three). In this paper, we discuss the application of a Mistral 7B model to address the Subtask one in English about finding the persuasive strategy that a meme employs from a hierarchy of twenty based just on its textual content. Only a portion of the reward is awarded if the technique's ancestor node is chosen. This classification issue is multilabel hierarchical. Our approach based on the use of a Mistral 7B model obtains a Hierarchical F1 of 0.42 a Hierarchical Precision of 0.30 and a Hierarchical Recall of 0.71. Our selected approach is able to outperform the baseline provided for the competition. 

\end{abstract}

\section{Introduction}


When information is intentionally crafted to serve a predetermined agenda, we often classify it as propaganda \cite{Geissler2023}. Propaganda employs various psychological and rhetorical strategies to achieve its goals \cite{cakmak2023283}. These methods encompass the utilization of logical fallacies and the manipulation of audience emotions \cite{Soares2023}. Logical fallacies can be particularly deceptive as they may initially appear sound and impartial, yet upon closer examination, it becomes evident that the conclusion cannot be logically derived from the premises. Another tactic involves employing emotionally charged language to sway the audience's opinion, bypassing rational analysis in favour of an emotional connection. Memes, typically comprising images overlaid with text, serve as a platform for propagandistic dissemination. Within deceptive memes, images either reinforce or complement textual techniques, or they themselves convey persuasive strategies.


To address these objectives, there is an ongoing demand for automated tools capable of extracting and categorizing data from online sources, facilitating the response to both established and emerging societal concerns. Recent advancements in machine and deep learning architectures have spurred heightened interest in Natural Language Processing (NLP). Substantial endeavours have been directed towards devising techniques for the automated identification and categorization of textual content accessible on the internet today. In the literature, to perform text classification tasks, several strategies have already been proposed. In the last fifteen years, some of the most successful strategies have been based on SVM \cite{colas2006comparison,croce2022svm}, on Convolutional Neural Network (CNN) \cite{kim2014convolutional,siino2021detection}, on Graph Neural Network (GNN) \cite{lomonaco2022courage}, on ensemble models \cite{miri2022ensemble,siino2022t100} and, recently, on Transformers \cite{vaswani2017attention, siino2022mcrock}.

The surge in the adoption of Transformer-based architectures within academic research has been further propelled by diverse methodologies showcased at SemEval 2024. These methodologies address a range of tasks and yield notable outcomes. For instance, in Task 2 \citep{jullien-etal-2024-semeval}, T5 is utilized to confront the challenge of identifying the inference relation between plain language statements and Clinical Trial Reports \cite{siino2024t5medical}. In Task 10, a Mistral 7B model is employed to perform emotion Recognition in Conversation (ERC) within Hindi-English code-mixed conversations \cite{siino2024trans}. Additionally, in Task 8 \citep{wang2024semeval}, a DistilBERT model is leveraged to identify machine-generated text \cite{siino2024badrock}.

Finally, for the Task 4 at SemEval 2024 \cite{semeval2024task4} – Multilingual Detection of Persuasion Techniques in Memes – three Subtasks were proposed. As already stated, in a disinformation campaign, memes effectively manipulate users through various rhetorical and psychological strategies, including causal oversimplification, name-calling, and smear tactics. The objective of this shared task is to develop models capable of detecting these techniques within the textual content of memes alone (Subtask 1), as well as within a multimodal framework where both textual and visual elements are analysed jointly (Subtasks 2 and 3). To face with the first Subtask in English, we proposed a Transformer-based approach which made use of Mistral 7B \cite{jiang2023mistral}. We used the model in a particular few-shot way described in the rest of this paper. Specifically, we provided the definitions of the 20 techniques to the model to identify, given each sample, all the techniques detected. We opted for Mistral 7B because the comparative analysis between Mistral 7B and other leading models, namely Llama 2 and Llama 1, reveals noteworthy advancements in common NLP tasks. Across multiple benchmark evaluations, Mistral 7B consistently exhibits superior performance in comparison to Llama 2, a prominent open 13B model. Moreover, its efficacy extends beyond mere parity with, but rather exceeds, the achievements of Llama 1, a state-of-the-art 34B model, particularly in tasks pertaining to reasoning, mathematics, and code generation. These findings underscore Mistral 7B's substantive contributions to the advancement of NLP, suggesting its potential as a benchmark model in the field.



The rest of the paper is made as follows. In Section \ref{sec:background} we provide some background on the Task 4 hosted at SemEval 2024. In Section \ref{sec:system_overview} we provide a description of the models presented. In Section \ref{sec:experimental_setup} we provide details about the experimental setup to replicate our work. In Section \ref{sec:results}, the results of the official task and some discussions are provided. In section \ref{sec:conclusion} we present our conclusion and proposals for future works.


We make all the code publicly available and reusable on GitHub\footnote{\raggedright\url{https://github.com/marco-siino/SemEval2024/}}.

\section{Background}
\label{sec:background}


This section furnishes background information regarding Task 4 (Subtask 1), held at SemEval 2024. The task entails identifying, based solely on the textual content of a meme, which of the 20 persuasion techniques, organized hierarchically, are employed. The selection of an ancestor node of a technique warrants only partial reward. The task thus presents a hierarchical, multilabel classification challenge. The hierarchical structure is illustrated in the official task's page, with 22 techniques depicted, although "Transfer" and "Appeal to Strong Emotion" are excluded from Subtask 1. For comprehensive details, please refer to provided resources.


For all Subtasks, the annotations from the PTC corpus, comprising over 20,000 sentences, were utilized where feasible. Although the corpus pertains to news articles, annotations adhere to identical guidelines, albeit with fewer techniques considered. As highlighted by the task coordinators, certain meme content may be deemed offensive or excessively potent by certain audiences. A similar multilingual corpus was also accessible during SemEval 2023 \cite{piskorski-etal-2023-semeval}. Here again, the corpus revolves around news articles across nine languages, yet the number of techniques and annotation guidelines differ marginally. A training set for local system development was additionally provided. Furthermore, the organizers furnished a development set and a public leaderboard for real-time result sharing among task participants. Ultimately, the organizers supplied a test set devoid of annotations and an online submission platform to evaluate the system performance.


Subtask 1 relies on the textual content extracted from memes as input data. Training, development, and test sets for all Subtasks are disseminated as JSON files, with each Subtask having its own individual file. For Subtasks 2a and 2b, in addition to the meme's textual content, input data includes the meme's image. In the Figure \ref{fig:sample}, is reported a sample from the official competition website\footnote{\url{https://propaganda.math.unipd.it/semeval2024task4/}}.

\begin{figure}
    \centering
    \includegraphics[width=1.0\linewidth]{Screenshot 2024-02-15 alle 08.56.23.png}
    \caption{An example from the dataset. In this case, two labels are assigned to the sample's text.}
    \label{fig:sample}
\end{figure}


Given the Figure \ref{fig:sample}: 

\begin{itemize}
    \item \textit{ID} consists of a unique identifier of the example across all the Subtasks;

\item \textit{text} represents the textual expression within the meme, formatted as a singular UTF-8 string. Initially, this text is automatically extracted from the meme, subsequently undergoing manual post-processing to rectify errors and arrange it such that each sentence occupies a distinct row. Furthermore, segments of text originating from distinct regions within the image are demarcated by blank rows. Notably, Task 1 qualifies as an NLP endeavour, given that image input is absent;

\item \textit{labels} denotes a compilation of permissible technique names identified within the text. These labels serve as the gold standard and will solely be furnished for the training set. In this particular instance, two techniques were identified: "Loaded Language" and "Name calling/Labelling."
\end{itemize}






\section{System Overview}
\label{sec:system_overview}

Even if it has already been proved that the Transformers are not necessarily the best option for any text classification task \cite{siino2022fake}, depending on the goal some strategies like domain-specific fine-tuning \cite{sun2019fine, van2023vietnamese}, or data augmentation \cite{lomonaco2023text,mangione2022improving} can be beneficial in several applications.

Our approach is a few-shot one \cite{littenberg2022comparing} and make use of the above-mentioned Mistral 7B. Mistral 7B, a language model equipped with 7 billion parameters, is designed to excel in both performance and efficiency. Compared to the leading open 13B model (Llama 2), Mistral 7B demonstrates superior performance across all evaluated benchmarks. Moreover, it outperforms the top released 34B model (Llama 1) in tasks related to reasoning, mathematics, and code generation. The model leverages grouped-query attention (GQA) to expedite inference, along with sliding window attention (SWA) to efficiently process sequences of varying lengths while minimizing inference costs. Additionally, a fine-tuned variant, Mistral 7B – Instruct, tailored for adhering to instructions, surpasses the Llama 2 13B – chat model across both human and automated benchmarks. The introduction of Mistral 7B Instruct underscores the ease with which the base model can be fine-tuned to achieve notable performance enhancements. Notably, this variant lacks any moderation mechanisms. The Mistral 7B Instruct variant requires a specific input format, as stated below:

\begin{quote}
<s>[INST] Instruction [/INST] Model answer</s>[INST] Follow-up instruction [/INST]
\end{quote}

\textit{Instruction}, along with the following \textit{Model answer}, can be a single sample with the related label or a set of sample/label pairs (realizing, in this case, a few-shot use of the model). Then, \textit{Follow-up instruction} is the current sample for which the prediction has to be provided by the model.  
More specifically, given the 20 persuasion techniques in memes, we have prepared a text string containing the techniques and their definitions to provide context in the template ready. The definitions of the 20 techniques are provided by the task organizers\footnote{\url{https://propaganda.math.unipd.it/semeval2024task4/definitions.html}}. At this point, the full text containing the twenty definitions plus the sample to be classified were provided as prompt to Mistral.


Then the question provided as prompt to mistral was: \textit{"Given the above Definitions of the Persuasion Techniques, Identify the Persuasion Techniques used in the Sentence. Answer using ONLY one or more numbers in the range 1-20 separated by commas. No text nor other options are allowed."}.

To this request, the model replied with one or more techniques detected in the corresponding sample. So, as an example, to the sentence: "Happy April Fools Day -  - Ooop I mean: March Fools day" the model replied to the prompt with the numbers 3 and 5. These two numbers correspond to the technique 3 (i.e., \textit{Whataboutism}) and to the technique 5 (i.e., \textit{Obfuscation, Intentional vagueness, Confusion}). It is important to mention that we also tried to use the model in a zero-shot configuration. In this case, we just asked the model to pick one or more categories given a meme. Unfortunately, the model did not report one or more correct categories, while developing discussions as answers.


Finally, we collected all the predictions provided on the test set to into a JSON file with the required format to submit our predictions.


As noted in the recent study by \cite{siino2024text}, the contribution of preprocessing for text classification tasks is generally not impactful when using Transformers. More specifically, the best combination of preprocessing strategies is not very different from performing no preprocessing at all in the case of Transformers. For these reasons, and to keep our system fast and computationally light, we have not performed any preprocessing on the text. The low impact of the best preprocessing techniques - or combinations of techniques - using Transformers, as reported in the study, is due to several factors like preserving the quantity and the quality of the original information available.


\section{Experimental Setup}
\label{sec:experimental_setup}

We implemented our model on Google Colab. The library we used comes from Hugging Face and as already mentioned is Mistral 7B\footnote{\url{https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF}}. We employed the v0.2 iteration of Mistral 7B, which represents an enhanced version of the Mistral-7B-Instruct-v0.1 model. To harness the capabilities of instruction fine-tuning, prompts must be enclosed within [INST] and [/INST] tokens. Additionally, the initial instruction should commence with a sentence identifier. The next instructions should not. The assistant generation will be ended by the end-of-sentence token ID. We also imported the Llama library \cite{touvron2023llama} from \textit{llama\_cpp}. The library is fully described on GitHub\footnote{\url{https://github.com/ggerganov/llama.cpp}}. The dataset provided for all the phases are available on the Official Competition page. We did not perform any additional fine-tuning on the model. To run the experiment, a T4 GPU from Google has been used. After the generation of predictions, we exported the results on the format required by the organizers. As already mentioned, all of our code is available on GitHub.


\section{Results}
\label{sec:results}

The evaluation was done by submitting to the leaderboard the predictions provided by the model. Subtask 1 and 2a are reliant on a hierarchical structure. The gold label consistently corresponds to a leaf node within the Directed Acyclic Graph (DAG). However, any node within the DAG can serve as a predicted label:

\begin{itemize}
    \item If the prediction does not correspond to a leaf node and is an ancestor of the correct gold label, a partial reward is issued, with the reward magnitude contingent upon the distance between the two nodes. For instance, if the gold label is "Red Herring" and the predicted label is "Distraction" or "Appeal to Logic."
    \item If the prediction does not align with any ancestor node of the correct label, no reward is granted. For instance, if the gold label is "Red Herring" and the predicted label is "Black and White Fallacy" or "Appeal to Emotions." A graphical representation illustrating this concept is provided.
\end{itemize}

However, it's worth noting that the hierarchical structure can be disregarded by confining predictions solely to technique names. This approach renders the task analogous to SemEval 2023 Task 3 \cite{piskorski-etal-2023-semeval}.

An illustrative example of the evaluation function can be accessed online\footnote{\url{https://propaganda.math.unipd.it/semeval2024task4/data/hierarchy_evaluation.html}}. In this case, the Subtask consists of a hierarchical multilabel classification task. Drawing from the aforementioned figure depicting the hierarchy, any node within the DAG can be designated as a predicted label. The gold label consistently corresponds to a leaf node within the DAG. Hierarchical-F1, detailed in \cite{kiritchenko2006learning}, is employed as the official evaluation metric. 




In the Table \ref{tab:competition_results}, the results obtained by the first three teams and by the last one, as showed on the official page\footnote{\url{https://propaganda.math.unipd.it/semeval2024task4/SemEval2024task4_test.html}}, are reported. Compared to the best performing models, our simple approach exhibits some room for improvements, although it is able to outperform the baseline. However, it is worth notice that it required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by Google Colab.


\begin{center}
\begin{table*}[ht]
\begin{center}
\begin{tabular}{ cccc } 
    \hline
  & H-F1 & H-Prec & H-Recall \\ 
  \hline
  914isthebest (1) & 0.752 & 0.684 & 0.836 \\ 
  \hline
  BCAmirs (2) & 0.698 & 0.668 & 0.732 \\ 
   \hline
  OtterlyObsessedWithSemantics (3) & 0.697 & 0.648 & 0.755 \\
  \hline
   Mistral 7B (30) & 0.42 & 0.30 & 0.71 \\ 
   \hline
  BASELINE (31) & 0.369 & 0.477 & 0.300 \\ 
  \hline
  IIMAS1UTM1LaSalle (33) & 0.199 & 0.755 & 0.115 \\ 
  \hline
\end{tabular}
\end{center}
\caption{\label{tab:competition_results}Comparing performance on the test set for Subtask 1 in English. In the table are shown the results obtained by the first three users and by the last one. Furthermore, is included the result of the baseline considered and of our approach making use of Mistral 7B. In parentheses is reported the position in the official final ranking.}
\end{table*}
\end{center}



\section{Conclusion}
\label{sec:conclusion}


This paper presents the application of Mistral 7B-model for addressing the Task 4 at SemEval 2024. For our submission, we decided to follow a few-shot learning approach, employing as-is, an in-domain pre-trained Transformer. After several experiments, we found beneficial to build a prompt containing the definitions of the techniques in memes. Then we provide, as a prompt, the definitions together with a sample. The model was asked to select all the techniques detected in the sentence. The task is challenging, and there is still opportunity for improvement, as can be noted looking at the final ranking. Possible alternative approaches include utilizing the zero-shot capabilities of other models like GPT and T5, increasing the size of the training set by using further data, or directly integrating ontology-based domain knowledge differently than what has been proposed in our work. Further improvements could be obtained with a fine-tuning and modelling the problem as a different text classification task.
 Furthermore, given the interesting results recently provided on a plethora of tasks, also other few-shot learning \cite{wang2023few, maia2024transformers, siino2023profiling, meng2024hierarchical} or data augmentation strategies \cite{muftie2023indobert, siino2024backtranslate, tapia2020data, siino2023xlnet} could be employed to improve the results. Looking at the final ranking, our simple approach exhibits some room for improvements. However, it is worth notice that required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by Google Colab. Also, thanks to the proposed approach, we have been able to outperform the baseline provided by the task organizers.

\section*{Acknowledgments}
We extend our gratitude to the anonymous reviewers for their insightful comments and valuable suggestions, which have significantly enhanced the clarity and presentation of this paper.

% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}

%\appendix

%\section{Example Appendix}
%\label{sec:appendix}

%This is an appendix.

\end{document}
