\documentclass{article}
\usepackage{svg}
\usepackage{graphicx} % Required for inserting images
%extra packages
\usepackage{comment}
\usepackage{subcaption}
\usepackage{amsmath} 
\usepackage{hyperref}
\usepackage{cleveref}
\usepackage{float}
\usepackage{authblk}
%extra comm
\newcommand\luca[1]{\textcolor{blue}{Luca: #1}}
\title{SemEval 2024 - Themis Method Description}
\author[1]{Luca Zedda}
\author[1]{Alessandra Perniciano}
\author[1]{Andrea Loddo}
\author[1]{Cecilia Di Ruberto}
\author[1]{Manuela Sanguinetti}
\author[1]{Maurizio Atzori}

\affil[1]{Department of Mathematics and Computer Science, University of Cagliari, Via Ospedale 72, 09124, Cagliari, Italy}

\begin{document}

\maketitle
\section{Abstract}
The pervasive presence of cultural and media-influenced content on digital platforms necessitates a systematic examination to address societal challenges such as biases and discrimination. This paper introduces a novel and efficient approach for multimodal meme analysis, specifically targeting the identification of persuasion techniques embedded within memes. Our methodology integrates Large Language Models (LLMs) and contrastive learning image encoders to discern the presence of persuasive elements in memes across diverse platforms. By capitalizing on the contextual understanding facilitated by LLMs and the discriminative power of contrastive learning, our framework provides a robust solution for detecting and classifying memes with persuasion techniques. 

\section{Challenge and Goal of the Experiments} 
The \href{https://propaganda.math.unipd.it/semeval2024task4/index.html}{SemEval 2024 Task 4} focuses on "Multilingual Detection of Persuasion Techniques in Memes." Memes, prevalent in online disinformation campaigns, wield significant influence, particularly on social media platforms. The task aims to develop models capable of identifying rhetorical and psychological techniques employed in memes to sway audiences. We focus this paper on subtask 2b in which participants must address a binary classification problem: determine whether a meme, comprising both textual and visual components, contains at least one persuasion technique among the predefined set of 22 techniques. 

\section{Model Description} 

\begin{figure*}[ht]
  \centering
  \includesvg[width=0.75\textwidth]{res/themis2}
  \caption{Themis model architecture}
  \label{fig:schema:arch}
\end{figure*}
The proposed method, named Themis, is a novel neural network architecture designed to analyze multimodal data, specifically targeting memes that often contain both textual and visual elements. Themis integrates both image and language models to process and understand the semantics embedded within memes effectively.

\subsection{Model Composition}
Our Themis model is composed of several interoperating components, in this section we review their function within the architecture.
\textbf{Image Embedding Model:} Themis incorporates an image embedding model to extract features from meme images. This model is responsible for processing pixel values and extracting meaningful representations from images. We opted for a pretrained CLIP~\cite{CLIP} encoder as our main encoder.

\textbf{Image Embedding Projection:}
In Themis, an image embedding projection is applied to the features extracted from the image embedding model. This projection serves as a method to project image features in LLM-compatible size.

\textbf{Language Model:} Themis utilizes a language model to handle textual and visual inputs associated with memes. This language model is trained to understand and generate text, providing a contextual understanding of the meme's content.

\textbf{Token Merger Module:} Themis employs a Token Merger module to merge tokens of image and textual features and attend to relevant information within the image. This allows the model to focus on salient aspects while processing memes.

\textbf{Classification Head:}

Themis incorporates a classification head to predict whether a meme contains specific persuasion techniques. This head takes the fused multimodal features and generates predictions based on the learned LLM representations.

\subsection{Model Freezing and LoRa Weights}

The Themis model utilizes freezing techniques to control the training of certain parameters. Specifically, both the image embedding model and the language model are frozen during training. This ensures that the pre-trained weights of these models are not updated, preserving the learned representations.

Additionally, Themis employs LoRa~\cite{LORA} weights to enhance its capabilities. LoRa weights are incorporated into the Image Embedding Projection layer and LLM model to introduce long-range interactions between tokens and patches, facilitating the capture of global context and improving overall performance in meme analysis tasks.

\section{Experiment Setup}
The experiments were executed on a workstation featuring an Intel Core i7-12700 @ 2.1GHz CPU, 32 GB RAM, and an NVIDIA RTX3060 GPU with 12GB of memory.
Among the different experiments, one main issue is denoted by the limited availability of VRAM; this issue not only limited our approach to smaller LLM and Image encoders but also limited batch size. Our experiments aim to enable efficient prediction even in such low-end system requirements. Table~\cref{tab:list_llm_enc} depicts the full set of selected Image encoders and LLM's that suited our requirements.

\input{texs/list_LLM_ENC}

To select our best model, we performed an intensive search over the best set of hyperparameters. Specifically, we varied the rank of the LoRa weight matrices (LoRa R), their alpha regularization factor (LoRa Alpha), the dropout rate of the LoRa weights (LoRa Dropout), and most importantly, we controlled the number of tokens by a Token Merging strategy.

\section{Experimental Evaluation}
In this section, we present the results of our experiments over the challenge development set. Table~\cref{tab:results} depicts the results of the different Image Encoders and LLMs by setting the LoRa R and LoRa Alpha both to 8, the LoRa Dropout to 0.2, and not performing any token merging. The results show that the use of larger Image Encoders such as CLIP Large increases the performances by an average of $0.7\%$ in terms of F-Measure. This could be explained by higher-dimensional embedding compared to the Base counterpart, but it also produces a larger number of tokens due to a smaller patch size.

Among the different LLMs, both TinyLlama and Phi-2 achieved the same performances. We selected TinyLlama for an extensive ablation study depicted in Table~\cref{tab:ablation} due to its limited number of parameters compared to the even-performing counterpart.

The ablation study results show that strong token merging strategies improve the model stability but limit its performances. The increase of the LoRa R greatly increases model instability due to the improved overfitting risk, while the increase in LoRa dropout greatly improves model performances, reaching the best F-1 result of $83.4\%$.

\input{texs/results}
\input{texs/ablation}

\section{Conclusions}
In this study, we introduced Themis, a novel model for analyzing multimodal memes by integrating Large Language Models (LLMs) and contrastive learning image encoders. Through comprehensive experiments, Themis demonstrated remarkable efficacy in detecting persuasion techniques within memes, achieving a notable F1 score of up to 83.4\%. Our findings underscore the critical role of meticulous model architecture design and hyperparameter optimization in meme analysis tasks. Notably, Themis presents a robust solution to combat societal challenges posed by biased content online, offering a promising avenue for mitigating the spread of misinformation and promoting digital discourse integrity.
\bibliographystyle{ieeetr}
\bibliography{biblio} % Entries are in the refs.bib file
\end{document}
