% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{tabularx}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{adjustbox}
\usepackage{subcaption}
\usepackage{color}
\usepackage{url}
\usepackage{lipsum}
\newcommand\blfootnote[1]{%
\begingroup
\renewcommand\thefootnote{}\footnote{#1}%
\addtocounter{footnote}{-1}%
\endgroup
}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{LMEME at SemEval-2024 Task 4: Teacher Student Fusion - Integrating CLIP with LLMs for Enhanced Persuasion Detection}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}
\author{Shiyi Li\authornote{*}, Yike Wang\authornote{*}, Liang Yang\authornote{†}, Shaowu Zhang, Hongfei Lin\\
  Dalian University of Technology \\
  \texttt{\{lishiyieee, yike\}@mail.dlut.edu.cn,\{liang, zhangsw, hflin\}@dlut.edu.cn} \\}

\begin{document}
\maketitle
\blfootnote{*These authors contributed equally to this work.}
\blfootnote{†Corresponding author.}
\begin{abstract}
This paper describes our system used in the SemEval-2024 Task 4 Multilingual Detection of Persuasion Techniques in Memes. Our team proposes a detection system that employs a Teacher Student Fusion framework. Initially, a Large Language Model serves as the teacher, engaging in abductive reasoning on multimodal inputs to generate background knowledge on persuasion techniques, assisting in the training of a smaller downstream model. The student model adopts CLIP as an encoder for text and image features, and we incorporate an attention mechanism for modality alignment. Ultimately, our proposed system achieves a Macro-F1 score of 0.8103, ranking 1st out of 20 on the leaderboard of Subtask 2b in English. In Bulgarian, Macedonian and Arabic, our detection capabilities are ranked 1/15, 3/15 and 14/15.


\end{abstract}

\section{Introduction}
Memes are one of the most popular content types in online disinformation activities. They thrive on social media platforms, effortlessly reaching vast audiences. Memes within disinformation activities employ various rhetorical and psychological techniques, such as oversimplification of causation, insults, and defamation, to achieve their impact on users. In this context, meme detection is crucial for identifying and reducing the spread of false information.

The SemEval-2024 Task 4 \citep{semeval2024task4} is a multilingual detection task involving persuasion techniques in memes, and we participate in Subtask 2b, the binary classification task, to identify whether it contains a persuasion technique or no technique.In addition to English, the task also includes three test datasets in different languages, which are only released together with the test data in the final phase of the task.The purpose of this design is to evaluate the model's performance in zero-shot scenarios, specifically its capability on languages it has not encountered before.

The key to meme detection lies in uncovering rich correlations within memes between seemingly unrelated text and image components, particularly when there is no apparent connection between the text and image. In cases where the implicit meaning needs deeper exploration and understanding, traditional detection methods often fall short, as they approach meme detection in a straightforward end-to-end manner, overlooking a profound comprehension of meme text and images. Recently, Large Language Models (LLMs) have found success in complex reasoning. They could reveal the underlying implicit meanings beneath the surface of memes, enabling the assessment of whether persuasion techniques are present. Inspired by heuristic teaching \citep{pintrich1996motivation}, where a teacher with rich experience can impart to students correct thinking and reasoning based on questions and corresponding answers, the students then learn how to deduce their own ways to the correct answers from questions. 

To better harness the powerful reasoning capabilities and knowledge reservoir of LLMs, we proposed a Teacher Student Fusion detection system based on the CLIP \citep{clip} model and LLMs. This system operates in two stages: in the first stage, as the teacher model, the LLM is used to extract prior background knowledge related to persuasion techniques from memes; in the second stage, leveraging this prior knowledge, we fine-tune a smaller student model to detect whether memes contain persuasion techniques.

\section{Related Work}

\subsection{Meme Classification Methods}
Meme classification has emerged as a rising multimodal task in recent years. Early multimodal approaches include models like concatBERT \citep{kiela2019supervised}, which simply concatenates features from both images and text. \citet{li2019visualbert} introduce the VisualBERT model, touted as the first image-text pretraining model. It utilizes Faster RCNN \citep{girshick2015fast} for image feature extraction, combines the extracted image features with text embeddings, and then inputs the concatenated features into a single Transformer structure initialized by BERT \citep{devlin2018bert} for classification. \citet{lee2021disentangling} propose the DisMultiHate model, which enhances the classification capability and interpretability of hate memes by introducing entity detection in memes and incorporating statistics on race and gender information as supplementary data. \citet{zia2021racist} employ the CLIP encoder to obtain features from both images and text, then simply concatenate these features and pass them to a logistic regression classifier. However, current solutions only capture the superficial signals of different modalities in memes in an end-to-end manner, failing to guide the model in-depth understanding of the complex and diverse relationships between visual and textual elements.

\subsection{Large Language Models}
Recently, LLMs \citep{brown2020language} have demonstrated remarkable reasoning capabilities, generating high-quality reasoning steps to augment input prompts to LLMs and improve their few-shot or zero-shot performance \citep{wei2022chain, kojima2022large, wang2022self}. Reasoning steps have also been employed for additional finetuning to "self-improve" LLMs \citep{zelikman2022star, huang2022large}. Unfortunately, the large size of LLMs restricts their deployment on detecting memes with diverse modalities. Knowledge distillation has been successfully used to transfer knowledge from larger, more competent teacher models into smaller student models affordable for practical applications \citep{buciluǎ2006model, hinton2015distilling, beyer2022knowledge}. However, existing researches on knowledge distillation from LLMs \citep{wang2022pinto, ho2022large, magister2022teaching} only consider the language modality. To accommodate multimodal features, we conduct abductive reasoning from LLMs, extracting underlying rationales as prompt arguments to assist in meme detection when fine-tuning smaller language models (LMs) for meme detection.
\begin{figure*}[!t]
\centering
\includegraphics[width=\textwidth]{model.pdf}
\caption{The framework of our method involves two stages. We first conduct abductive reasoning from the teacher model to extract rationales (\textcolor[RGB]{184,179,201}{purple}) by the prompt consisting of the meme text (\textcolor[RGB]{148,197,233}{blue}), the image caption (\textcolor[RGB]{255,227,144}{yellow}), and the label (\textcolor[RGB]{170,229,143}{green}). Utilizing the generated rationales, we then train the student model and predict whether memes contain persuasion techniques.}
\label{fig:1}
\end{figure*}
\section{System overview}
We define a meme detection dataset that potentially contains persuasion techniques as a set of memes where each meme $M = \left\{ I,T,y \right\}$ is a triplet representing a visual content $I$ that is associated with the textual $T$, and a ground-truth label $y \in \left\{ propagandistic, ~non\_ propagandistic \right\}$.

The core idea of our teacher student fusion model is to reason and develop a cognition-level rationale beyond the recognition-level perception \citep{davis2015commonsense} by constraining the relationships between visual and textual elements in memes. To better utilize multimodal reasoning distilled from LLMs, this task is formulated as a natural language generation paradigm, where our model takes the text $T$ and image $I$ as the input and generates a textual output of the label $y$ to clearly express whether at least one persuasion technique is present in the meme or not. In this paper, we propose to utilize abductive reasoning from LLMs with multimodal inputs to train smaller downstream models. Our overall framework is illustrated in Figure \ref{fig:1}, which consists of abductive reasoning from LLMs and model fine-tuning.

\subsection{Stage 1: Abductive Reasoning from the Teacher Model}
We activate explicit reasoning knowledge in LLMs as the teacher model. Through prompt learning in causal reasoning, the LLM acquires meme-related context and hidden information, to guide student model in detecting persuasion techniques. Given a meme sample $M$ from the training data, we first extract the text caption $\hat{I}$ of the image $I$ to represent the visual content by off-the-shelf captioning model\footnote{\url{https://huggingface.co/nlpconnect/vit-gpt2-image-captioning}}. Specifically, based on the triplet $\left\{ \hat{I},T,y \right\}$, we design a prompt $p$:

"\emph{Given the textual of a meme: [$T$], which is embedded in its image: [$\hat{I}$], labeled as [$y$]. Please provide a streamlined rationale for inferring the meme as [$y$], incorporating prior background knowledge related to persuasion techniques but without explicitly indicating the label.}"

It prompts the LLMs to generate a rationale $R$ including rich contextual background knowledge, enabling the inference of whether persuasion techniques are present in memes.

\subsection{Stage 2: Student Model Fine-tuning}
To facilitate the interaction between meme text and images, we fine-tune a smaller student model for persuasion detection tasks. By using the reasons generated by the teacher model as background knowledge, we aid in uncovering the rich interrelationship between text and vision modalities of memes. For a meme sample \(M\),we first use the encoder of the CLIP model to encode the text \(T\) and image \(I\) input to obtain the embedding vector \(H_T^\) and \(H_I^\).The advantage of attention mechanism in modality fusion and alignment lies in its ability to dynamically allocate and adjust weights for different modalities, allowing the model to flexibly focus on specific parts of the input. By emphasizing relationships between modalities, the attention mechanism helps improve the effectiveness of modality fusion and enhances the model's ability to capture important information. Therefore, we adopted a cross-attention mechanism for the fusion of textual and visual features.
\begin{equation}
Q_T=W_QH_T+b_Q
\end{equation}
\begin{equation}
K_I=W_K\ H_I+b_K
\end{equation}
\begin{equation}
V_I=W_V\ H_I+b_V
\end{equation}

\begin{equation}
\text{H}_o &= \text{Softmax}\left(\frac{(K_I)^T Q_T}{\sqrt{d}}\right) V_I
\end{equation}
Among them, $d$ is the dimension of the feature, softmax is the activation function.

\paragraph{Learn from LLM}By inputting the fused features \(H_o\) to the transformer decoder for decoding,we obtain the decoded output. Subsequently, we calculate the cross-entropy loss between this output and the given reason, as expressed by the following formula. Minimizing the cross-entropy loss between the meme feature and the generated reason feature facilitates the extraction of prior knowledge from the generated reason. This process helps the model transfer the contextual background information from the generated reason to the meme feature.
\begin{equation}
\mathcal{L}_{llm}=CrossEntropy(decoder(H_o),R)
\end{equation}

\paragraph{Label Training}Through the aforementioned process, our model has acquired the reasoning ability to extract persuasion techniques from LLM. As the objective of the task is to determine whether a meme contains persuasion techniques, label prediction becomes essential. This process shares the same model architecture as the preceding steps, with the introduction of a classifier during the decoding phase for label prediction. During the training process, we optimize the model by minimizing the cross-entropy loss between the predicted labels and the ground truth labels. The loss function is expressed as follows.
\begin{equation}
\mathcal{L}_{label}=CrossEntropy(y^{pre},y^{true})
\end{equation}

Through the above process, for the data samples to be predicted, we can directly predict the labels of the model without generating corresponding rationales.

\section{Experiments}

\subsection{Dataset and Evaluation}
The dataset from the Task 4 of SemEval-2024 contains memes potentially employing persuasion techniques. After training on the English dataset, the model is tested across four languages: English, Bulgarian, North Macedonian and Arabic. As mentioned in the official task description, we employ macro-F1 to evaluate the performance of binary classification Subtask 2b.

In the experiments, we consider the 7b LLaMa2 model \citep{llama} as the teacher model. For the task-specific student model, we utilize the CLIP-ViT-B/32 \citep{clip} architecture as the foundational framework. During the training phase, we evaluate the performance of the model every 100 steps and retain the parameters of the model that performed best on the validation set. The hyperparameters settings adopted are detailed in Table~\ref{tab:setting}. All models are trained on NVIDIA GeForce GTX 3090 GPU.

\begin{table}
\centering
\begin{tabular}{ccc}
\toprule[1pt]
Parameter & From LLM    & Label Training \\
\midrule[0.7pt]
Epochs          & 20                & 10\\
Batch size      & 32                & 32\\
Learning rate   & 5e-4              & 5e-5\\
Warmup step     & 0.1               & 0.1\\
Warmup Strategy & Linear            & Linear\\
Image size      & 224               & 224\\
\bottomrule[1pt]
\end{tabular}
\caption{Hyper-parameters.}
\label{tab:setting}
\end{table}

\begin{table}
\centering
\begin{tabular}{ccc}
\toprule[1pt]
Dataset        & Model          & Macro-F1 \\
\midrule[0.7pt]
               & baseline        & 0.2500   \\
English - Dev  & LMEME(w/o llm) & 0.8329   \\
               & LMEME          & 0.8428   \\
\midrule[0.7pt]
               & baseline        & 0.2500   \\
English - Test & LMEME(w/o llm) & 0.8043   \\
               & LMEME          & 0.8103   \\
\bottomrule[1pt]
\end{tabular}
\caption{Main experimental results of Subtask 2b in English. LMEME is the model proposed in our study. LMEME(w/o llm) represents the ablation results without rationales generated by the teacher model. The baseline refers to the evaluation's benchmark.}
\label{tab:accents}
\end{table}


\subsection{Results and Analysis}
Table~\ref{tab:accents} shows the English detection capabilities of our system in Persuasion Techniques in Memes. A noticeable improvement is observed when comparing it to the scenario without the incorporation of prior knowledge from the LLM, our system in this study demonstrates superior performance. This indicates that leveraging background knowledge obtained from the teacher model can enhance the model's understanding of persuasion techniques to some extent, assisting the model in more accurately detecting memes. Furthermore, the combination of CLIP's encoding capability and the design of cross attention fusion using attention mechanisms enables the system to better align semantic features between text and visuals, facilitating more effective meme persuasion detection.


\begin{table}
\centering
\begin{tabular}{ccc}
\toprule[1pt]
Dataset                 & Model          & Macro-F1 \\
\midrule[0.7pt]
                        & baseline       & 0.1667   \\
Bg - Test        & LMEME(w/o llm) & 0.6250   \\
                        & LMEME          & 0.6710   \\
\midrule[0.7pt]
                        & baseline       & 0.0909   \\
NM - Test & LMEME(w/o llm) & 0.5536   \\
                        & LMEME          & 0.5908   \\
\midrule[0.7pt]
                        & baseline       & 0.2271   \\
Ar - Test           & LMEME(w/o llm) & 0.2933   \\
                        & LMEME          & 0.3620   \\
\bottomrule[1pt]
\end{tabular}

\caption{Main experimental results in Bulgarian, North Macedonian
and Arabic.}
\label{tab:otherlanguage}
\end{table}

Table~\ref{tab:otherlanguage} displays our system's zero-shot capability in Bulgarian, North Macedonian, and Arabic. It can be observed that our system exhibits a significant improvement over baseline results in Bulgarian and North Macedonian, ranking 1/15 and 3/15 in the task. In comparison, there is also an improvement in results for Arabic, although not as pronounced as in the first two languages. The potential reason for this lies in the fact that English is an inflectional language with some agglutinative and analytic features. Bulgarian and Macedonian also share some features as inflected languages. However, Arabic introduces agglutinative features onto its inflectional foundation. Since our model has been trained on an English dataset, its effectiveness in detecting agglutinative features might be slightly inferior compared to inflectional languages. Additionally, the ablation results on these three datasets also indicate the superiority of introducing prior knowledge. This further validates the effectiveness of the teacher student fusion system proposed in this paper.


\section{Conclusion}

This paper provides a detailed exposition of our approach in addressing Subtask 2b of Semeval2024 Task 4. Our teacher student fusion system initially leverages the Large Language Model as the teacher model to generate background knowledge regarding whether memes contain persuasion techniques. Subsequently, we incorporate this knowledge to fine-tune the student model by minimizing cross-entropy loss, sharing learned parameters with the predictive parameters of the model. Finally, we proceed with predictions using the trained model.

In the future, we will explore alternative encoding methods, better aligned image and textual semantic fusion methods, LLMs of different sizes and types, and different ways of prompting LLMs to generate enhanced background knowledge for meme persuasion detection.
% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}

\appendix
\end{document}
