
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt,table,xcdraw]{article}

% Remove the "review" option to generate the final version.
%\usepackage[review]{ACL2023}
\usepackage{ACL2023}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
\usepackage{multirow}
\usepackage{xcolor}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}


% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Hidetsune at SemEval-2024 Task 4: An Application of Machine Learning to Multilingual Propagandistic Memes Identification Using Machine Translation}

% Author information can be set in various styles:
% For several authors from the same institution:
\author{{Hidetsune Takahashi}\\
        Waseda University \\ takahashi78h@toki.waseda.jp} %\\ Nishiwaseda 1-6-1, Tokyo, JAPAN}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

% \author{First Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\\And
%   Second Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\}

\begin{document}
\maketitle
\begin{abstract}
In this system paper for SemEval-2024 Task 4 subtask 2b, I present my approach to identifying propagandistic memes in multiple languages. I firstly establish a baseline for English and then implement the model into other languages (Bulgarian, North Macedonian and Arabic) by using machine translation. Data from other subtasks (subtask 1, subtask 2a) are also used in addition to data for this subtask, and additional data from Kaggle are concatenated to these in order to enhance the model. The results show high reliability of my English baseline and a room for improvement of its implementation.  

\end{abstract}

%4 pages!
\section{Introduction}
%Understanding semantic relatedness between texts provides prospective development toward our understanding of how we construct meaning in our sentences and fills the gap between intended instructions and interpreted messages. Through this deconstruction of natural language, we can find that there is a bonafide application that stretches across different fields and allows us to garner insight into languages’ semantics and use this knowledge to build a bridge between context and its relevance in textual linguistics.

SemEval 2024 Task 4 \citep{semrel2024task} calls for classification of memes into different persuasion techniques in textual content only (subtask 1) or in textual and visual content (subtask 2a), and identifying whether or not memes are propagandistic (subtask 2b). I participate in subtask 2b, which is a binary classification problem between "propagandistic" and "non\_propagandistic". Various memes are provided in English, Bulgarian, North Macedonian and Arabic to determine whether the memes are propagandistic or not.

% Due to the expansion of social media use, there have been more and more memes on the internet that intend to influence people's political thoughts so that they become for or against particular candidates or political parties. Therefore, classification discussed in this subtask might become a more and more important topic in the near future for people to recognize correct information to make appropreate decisions on politics.

My baseline is established so that it achieves fairly high accuracy in English. Although it adopts a classical machine learning method with training, the training data are adjusted by being concatenated with additional dataset. After setting up my baseline, the model is implemented into other languages using machine translation. 

Participating in this task allows me to test the ability of my model, achieving a fairly high score for its simplicity. In future studies, the model can be strengthened well enough with appropriately adjusted training data. As for the other languages, on the other hand, the implementation of my English baseline does not necessarily show consistent reliability. Although my baseline works relatively well for Arabic to some extent, the scores go down drastically for Bulgarian and North Macedonian. One of the main reasons for this issue might be that accuracy of machine translation is not high enough, changing the original meanings of the memes and possibly making it more difficult for the English based model to identify propagandistic memes.



My code is available on GitHub \footnote[1]{\url{https://github.com/Hidetsune/SemEval2024_Task4.git}}.




\section{Background}
%SEPARATION OF TWO LANGUAGES
%OTHER WAYS TO DEAL WITH TWO-LANGUAGE MIXED SENTENCES
%ABOUT HINDI CHARACTER
%SPACY IS WELL SUITED FOR ENGLISH
%WHY IS THIS TASK IMPORTANT?
%WHRITTEN FORM OF HINDI VS PRONUNCIATION FORM OF HINDI
%HINGLISH

The subtask I participate in focuses on classification of memes. Given a json file that has an ID, an image name and text for each meme as input, the subtask requires assigning either "propagandistic" or "non\_propagandistic" to the memes. In development phase, data in English were given and participants were allowed to test their solutions for the English data only. Participants were told that they would also have unlabeled test data in three non-English languages as the evaluation phase starts, meaning that no information was released about non-English languages in the development phase.   

Propagandistic memes on social media have become a growing issue in the past few years. As more and more people use social media platforms, many types of information including propagandistic one is spread to a number of people \citep{bradshaw2019global}. These days, issues caused by propaganda on social media have become worse than most people might think, as a study by \citet{o2019social} gives a warning. In fact, these memes did change people's thoughts for elections influencing people's voting behaviors \citep{aral2019protecting}. Therefore, it might become more and more important in the future for NLP to be able to recognize whether or not the information uses persuasion techniques with high accuracy. 

There are already many previous studies that adopt modern NLP techniques for propagandistic memes detection. A previous study by \citet{9811117} uses RoBERTa, which is the state-of-the-art pre-trained language model at the time, resulting in a F1 score of 60.2\%. Another previous study by \citet{sprenkamp2023large} tries modern Large Language Models including GPT-3 and GPT-4, also resulting in reliable scores. \citet{al-omari-etal-2019-justdeep} utilizes combinations of multiple deep learning models including BERT, BiLSTM and XGBoost with accuracy of around 0.67 in F1 score.

On the other hand of these previous studies, my methodology intends to test classical machine learning approach rather than state-of-the-art LLMs. During my methodology, SpaCy-v3 model \citep{mymodel} is trained with over 20000 memes in English, showing high reliability with accuracy of 0.71353 in F1 macro. Then, the trained model is implemented into non-English languages (Bulgarian, North Macedonian and Arabic) by using Google Translate \citep{googlemachinetranslation}.

My participation in this task reveals a high potential of applying classical NLP methods to detection of propagandistic memes with properly adjusted training data. The results reveal that a fairly high score can be achieved without state-of-the-art LLMs and complicated methods. Although the direct implementation of my English baseline into other languages has a room for improvement, the accuracy might go up easily with better machine translation models and careful consideration of differences in topics behind the memes. This paper introduces both strengths and weaknesses of my approach, guiding a direction to future application of classical machine learning to a modern issue of propagandistic memes that requires automatic binary classification.



\section{System overview}

The main strategy of my system is a classical machine learning method for English baseline and implementation of it into other languages using machine translation. A quick overview of my algorithm is as follows.\par

\begin{enumerate}
    \item {\textbf{Data preparation}}: Using official datasets from all the subtasks and additional data on Kaggle\footnotemark[2], training data are prepared to have 11001 memes (including 11000th counting from 0) in English for both propagandistic and non-propagandistic at maximum, making up nearly 22000 memes in total.
    \item {\textbf{Training}}: Train a SpaCy model \citep{mymodel} using the prepared training data. Both the training and test data are processed so that they do not have usernames that appear in the additional data and all the memes are lower cased for high efficiency to train the model.
    \item{\textbf{Implementation into non-English languages}}: Translate non-English memes into English. This process enables my established baseline to perform in multiple languages, and machine translation is used in this step. After translation of the memes, the model is used in the same way as in English memes.
\end{enumerate}

The system imports a prepared training CSV file as a pandas dataframe, where all the data from Task 4 (train and validation data of subtask 1, subtask 2a and subtask 2b) and additional data from Kaggle\footnote[2]{\url{https://www.kaggle.com/datasets/thoughtvector/customer-support-on-twitter}} are concatenated to compose a large training data with 20774 rows. Only first 20000 rows are used as for the additional data\footnotemark[2] due to its large data size, and all the rows that exceed the limitation of 11001 memes (for both "propagandistic" and "non\_propagandistic") are eliminated from the concatenated dataset. Memes in the data are all in English for the purpose of establishing a baseline that guarantees fairly high accuracy in English. Then, SpaCy-v3  model \citep{mymodel} is trained using the resulting data, and test data with unlabeled memes in English is imported as a json file. After that, the trained model is used to assign either "propagandistic" or "non\_propagandistic" to each unlabeled meme. As for other languages (Bulgarian, North Macedonian and Arabic), the memes are translated into English so that my English baseline can be implemented into them. Google Translate \citep{googlemachinetranslation} is adopted in this part, and the same trained model is used similarly to the English baseline after translation.

My participation in this task allows for testing the classical NLP approach and simple implementation of it using machine translation. The English baseline, which uses classical machine learning methods, achieves its certain ability to identify propagandistic memes with a reliable score. 

On the other hand, the scores go down as for non-English languages. This might be because of slight changes in meanings in the translation process, which turns out to be the biggest weakness of my approach. Classification of English memes is fairly difficult even for humans given a single utterance. For instance, "VOTE REPUBLICAN. THEY MAY NOT BE PERFECT. BUT THE OTHER SIDE IS INSANE." can be easily classified as propagandistic, but sentences like "CRY ALL YOU WANT...... HE'S DOING EXACTLY WHAT I HIRED HIM FOR......" might not be that easy for the classification because their nuance might depend on situations (on SNS or at work etc.) to some extent. Since propagandistic memes classification is difficult in this way, possible changes of original meanings by machine translation might have resulted in a serious issue for classification of test data, lowering the scores of other languages dramatically.

Another possible reason for the lowered performance in non-English languages is that there are large differences in topics. Words like "Trump" and "Russia" frequently appear in English memes, but "Bulgarian" is one of the most frequently used words in North Macedonian memes. Since Bulgaria and North Macedonia have some diplomatic issues \citep{kamberi2023overview}, memes that have basis on them  ("THE BULGARIANS ENTER THE CONSTITUTION" for example) tend to appear frequently. This kind of differences in topics probably caused the lowered accuracy in non-English languages, revealing a new challenging problem of the approach to implement my English baseline into other languages.    








%data set citation
%\cite{abdalla-etal-2023-makes}
%Requested citations
%\cite{asaadi2019big}
%\cite{abdalla2021makes}

%from papers with code
% \cite{wei2023synwmd}
% \cite{merkx2021semantic}
% \cite{hammerl2023exploring}
%\cite{BATET2016365}
%\cite{jeyaraj2021mnet}

\section{Experimental setup}
%WHICH OFFICIAL DATA DID I USE?
%WHY DID I DECIDE ON THAT ADDITIONAL DATA? (BECAUSE CUSTOMER SERVICE ON TWITTER ARE PROBABLY NOT ABOUT PROPAGANDISTIC TOPICS)
%HOW MANY TEXTS AS LIMITATION FOR EACH PROPA/NONPROPA? AROUND 11000

Before moving on to the actual training of the model, data preparation was an essential part of my solution. First of all, all the data of this task (including other subtasks) are imported as json files and converted into pandas dataframes. The dataframes have "text" and "propagandistic/non\_propagandistic" as columns. Training data and validation data from both subtask 1 and subtask 2 are composed of "propagandistic" only, so they are concatenated as "large\_data1", which is a dataframe with 8307 rows and all "propagandistic" memes. Also, data from Kaggle\footnotemark[2] are imported as all "non\_propagandistic" dataframe and processed so that it has no usernames in "text" column. Validation data of subtask 2b are previously imported, but I decided to increase weight of training data of task 2b by twice and not to use validation data after trial submissions in development phase. Therefore, the resulting training data are composed of training data from subtask 2b (weight increased by twice), "large\_data1" and additional data from Kaggle\footnotemark[2]. The training dataframe is adjusted so that it has 11001 memes for both propagandistic
and non-propagandistic at maximum, being shuffled to be ready for training.

The reason why the additional data from Kaggle\footnotemark[2] are chosen is that they are less likely to be propagandistic compared to many other existing datasets. The data are from customer support on Twitter including AppleSupport and AmazonHelp, so the topic there should be something related to their products or services. There are many other existing datasets extracted from social media platforms, but it takes too much time end effort to manually assign propagandistic and non-propagandistic to each utterances. For the purpose of getting non-propagandistic utterances only, it might be one of the easiest and most realistic approach to find an existing dataset whose topic is clearly unrelated to politics and diplomacy as included in my solution. 

After these data preparation steps, the model \citep{mymodel} is trained with the training dataset, and test data with unlabeled memes are imported as a json file. The memes in non-English languages are translated into English as stated in the previous section. The trained model assigns either "propagandistic" or "non\_propagandistic" to each meme. The training data and test data are cleaned with new line removal and lower casing prior to use of them.






\section{Results}

%MT ACCURACY
%I ONLY HAD MACEDONIAN FOR MT; NOT SPECIALIZED IN NORTH MACEDONIAN
%THE TOPICS IN EVAL DATA FOR OTHER LANGUAGES ARE DIFFERENT FROM THE ONES IN TRAIN DATA FOR ENGLISH(MAINLY DISCUSSING TRUMP); CONTROL+F IN EVAL_TASK4_NORTH MACEDONIAN TO SEARCH FOR "BULGARIAN" AS TAKEN IN PHOTOS



\begin{table}
    \centering
    \setlength{\tabcolsep}{3pt}%
    \begin{tabular}{cccc}\hline
         LANGUAGE&F1 macro&  F1 micro& Ranking \\ \hline\hline
         English&0.71353&  0.79000& 13/20 \\
         Bulgarian&0.32670&  0.33000& 14/14 \\
         North Macedonian& 0.38942&  0.46000& 13/14 \\
         Arabic&0.52825&  0.54375& 9/14 \\ \hline
    \end{tabular}
    \caption{Task scores for multiple languages}
    \label{tab:my_label}
\end{table}

Table 1 shows official results of my solutions. They show fairly high accuracy of my English baseline with nearly 0.8 in F1 micro. It can be said that my methodology for English baseline using classical machine learning works fairly well with a thoughtful training data adjustment. 

As for the non-English languages, the scores go down due to the potential reasons as stated in prior sections. Even so, Arabic has relatively high accuracy, which is lower than English by around 0.18 in F1 macro but higher than Bulgarian by around 0.20. This difference might have been caused by accuracy of machine translation mainly. Arabic is a widely used language with a total of about 372.7 million native speakers in the world.\footnote[3]{\url{https://www.worlddata.info/languages/arabic.php}} There is a possibility that Google Translate \citep{googlemachinetranslation}, which I use for machine translation, has higher accuracy for widely used languages including Arabic, maintaining the original meanings and nuances of the memes fairly correctly. 







\section{Conclusions}
%What else could we have done
%What worked and didn't work
%We started with the simplest approaches such as SpaCy similarity measures to add to the original baseline measures...


To summarise, my methodology firstly focuses on establishing a baseline that guarantee fairly high accuracy for English memes. After that, the baseline is implemented into non-English languages by translating the memes into English using machine translation.

The results show high reliability of my English baseline. The methodology for the baseline has its basis on classical machine learning, but my participation in this task reveals its fundamental abilities to deal with complicated classification task with properly adjusted training data.

On the other hand, the results also show that the simple application of my English baseline has a room for improvement. The scores of non-English languages dramatically dropped although Arabic has relatively reasonable accuracy compared to Bulgarian and North Macedonian. There can be many possible reasons for this including the accuracy of machine translation and changes in topics between memes in different languages.

In future studies, it might be worthwhile to enhance the model with many more memes for my English baseline. Collecting propagandistic memes might be a time consuming task, but non-propagandistic memes, on the other hand, can be easily found and used by utilizing existing datasets whose topics clearly have nothing to do with politics and diplomacy. As for non-English languages, higher accuracy might be achieved by using better machine translation models and enhancing the baseline model with specific political or diplomatic topics in the countries.




% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}


\end{document}
