
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt,table,xcdraw]{article}

% Remove the "review" option to generate the final version.
%\usepackage[review]{ACL2023}
\usepackage{ACL2023}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
\usepackage{multirow}
\usepackage{xcolor}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}


% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Hidetsune at SemEval-2024 Task 3: A Simple Textual Approach to Emotion Classification and Emotion Cause Analysis in Conversations Using Machine Learning and Next Sentence Prediction}

% Author information can be set in various styles:
% For several authors from the same institution:
\author{{Hidetsune Takahashi}\\
        Waseda University \\ takahashi78h@toki.waseda.jp} %\\ Nishiwaseda 1-6-1, Tokyo, JAPAN}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

% \author{First Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\\And
%   Second Author \\
%   Affiliation / Address line 1 \\
%   Affiliation / Address line 2 \\
%   Affiliation / Address line 3 \\
%   \texttt{email@domain} \\}

\begin{document}
\maketitle
\begin{abstract}
In this system paper for SemEval-2024 Task 3 subtask 2, I present my simple textual approach to emotion classification and emotion cause analysis in conversations using machine learning and next sentence prediction. I train a SpaCy model for emotion classification and use next sentence prediction with BERT for emotion cause analysis. While speaker names and audio-visual clips are given in addition to text of the conversations, my approach uses textual data only to test my methodology to combine machine learning with next sentence prediction. This paper reveals both strengths and weaknesses of my trial, suggesting a direction of future studies to improve my introductory solution.

\end{abstract}

%4 pages!
\section{Introduction}
%Understanding semantic relatedness between texts provides prospective development toward our understanding of how we construct meaning in our sentences and fills the gap between intended instructions and interpreted messages. Through this deconstruction of natural language, we can find that there is a bonafide application that stretches across different fields and allows us to garner insight into languages’ semantics and use this knowledge to build a bridge between context and its relevance in textual linguistics.

SemEval 2024 Task 3 \citep{wang2022multimodal} calls for assigning an emotion to each utterance and extracting its emotion cause in conversations. Subtask 2, which I participate in, requires emotion classification and identification of emotion cause utterances with audio-visual clips available whereas subtask 1 requires identification of specific textual span as well without audio-visual clips. 

I participate in subtask 2, for which a speaker name, text and an audio-visual clip are given for each utterance. Instead of not identifying specific cause span in the emotion cause utterance in this subtask, I set a limitation to use textual data only while audio-visual data are also available. Therefore, my methodology uses textual data of utterances only as input to classify emotions and identify cause utterance numbers as output. For this reason, training data from subtask 1, for which video names are not given, are used instead of data from the subtask I participate in.

While the task \citep{wang2022multimodal} prohibits use of additional annotation data, I overlooked the sentence stating the rule and mistakenly used additional data for my solution. I would like to show my appreciation for the task organizers and readers acknowledging and understanding my mistake of using additional data. 

For data preparation, official training data for subtask 1 (CSV converted version) \citep{wang2023multimodal}, training data (translated and CSV converted version) from SemEval2024 Task10 subtask 1 (ERC) by \citet{task10dataset} and data by \citet{additionaldata} are used. They all are concatenated in that order and adjusted so that the resulting dataset has first 7001 neutral utterances (including 7000th counting from 0) and first 5001 utterances at maximum for each emotion other than neutral. 

Then, SpaCy-v3 model \citep{mymodel} is trained using the adjusted training data for emotion classification. In addition to that, next sentence prediction \citep{nextsentenceprediction} is used for identification of emotion cause utterances. In that step, I decided to simplify the methodology by hypothesising that the emotion cause utterance is the utterance itself or its previous utterance. With this simple assumption, my algorithm checks the relatedness of each utterance and its previous utterance using next sentence prediction \citep{nextsentenceprediction}, which returns true or false. Previous utterance is chosen as cause utterance if the two utterances are deemed related, and the utterance itself if not.    

The result shows a limited performance of my introductory solution, but it also clarifies a direction to its improvement. Although my combined methodology has a large room for improvement, it does have a potential in its simplicity and limitation to use textual data only. This paper aims to share an experimental trial to test my combined methodology, guiding a direction to its future application and improvement.


My code is available on GitHub \footnote{\url{https://github.com/Hidetsune/SemEval2024_Task3.git}}.




\section{Background}
%SEPARATION OF TWO LANGUAGES
%OTHER WAYS TO DEAL WITH TWO-LANGUAGE MIXED SENTENCES
%ABOUT HINDI CHARACTER
%SPACY IS WELL SUITED FOR ENGLISH
%WHY IS THIS TASK IMPORTANT?
%WHRITTEN FORM OF HINDI VS PRONUNCIATION FORM OF HINDI
%HINGLISH

The subtask I participate in (subtask 2) focuses on emotion classification and emotion cause analysis with text data and audio-visual clips. Subtask 1, on the other hand, does not allow participants to use audio-visual clips and requires extracting specific textual cause spans as well. My participation in subtask 2 sets a limitation to use textual data only, which means that it is substantially the same as subtask 1 except that I do not extract specific textual cause spans. Training data from subtask 1 are used instead of that from subtask 2 because they seem to be identical to each other except that they have no video names in the dataset. Therefore, my methodology for subtask 2 uses data from subtask 1 and additional data from other sources for training. Given the evaluation dataset with audio-visual clips available, my methodology, which is trained by textual data only, assigns an emotion category and its cause utterance as output using textual data of the evaluation dataset.

This task is technically a mixture of two topics, which are emotion classification and emotion cause analysis. As for emotion classification, many previous studies have been conducted especially on social media including Twitter and Facebook. For instance, a work by \citet{gaind2019emotion} classifies text on social media into six emotion categories with high accuracy. Another study by \citet{brynielsson2014emotion} investigates in people's emotions during crises using a support vector machine. In addition to its use for social media, its application to real conversations is also getting an attention. A study by \citet{s21041322}, for example, applies emotion detection to social robotics, aiming to improve its ability to interpret feelings of humans from a viewpoint of NLP methods. 

There are many previous studies for emotion cause analysis too. A study by \citet{fan2019knowledge}, for example, uses hierarchical neural network to get high accuracy. Another study by \citet{ding2020end} adopts a complicated approach, resulting in reliable accuracy.

On the other hand, this paper aims to test a simple approach to combine classical machine learning method with next sentence prediction with a certain assumption. My methodology has a strength in its simplicity, but the result shows a large room for improvement.




\section{System overview}

The main strategy of my system is a combination of classical machine learning method with next sentence prediction. Machine leaning is used for emotion classification and next sentence prediction is used for identification of emotion cause utterances. Audio-visual clips are available in this subtask, but only textual data of the utterances are used for my solution.  A quick overview of my combined methodology is as follows.

\begin{enumerate}
    \item \textbf{Training data preparation}: Official training data from subtask 1 \citep{wang2023multimodal} are converted from a json file into a pandas dataframe. Similarly, training data from SemEval2024 Task10 subtask 1 (ERC) \citep{task10dataset} are translated into English and converted into a pandas dataframe. The converted dataframes and data by \citet{additionaldata} are concatenated to compose an adjusted training data. The adjusted data have two columns, in which text and an emotion are stored respectively for each utterance.
    
    \item \textbf{Emotion classification using machine learning}: Using the adjusted training data, SpaCy-v3 model \citep{mymodel} is trained and used for emotion classification. It assigns an emotion to each utterance from "neutral", "surprise", "anger", "sadness", "joy", "disgust" and "fear".
    \item \textbf{Emotion cause utterance identification using next sentence prediction}: If an assigned emotion is not "neutral", next sentence prediction \citep{nextsentenceprediction} identifies its emotion cause utterance. My methodology works under the simple assumption that emotion cause utterance is the utterance itself or its previous utterance.
    
\end{enumerate}




In the first step, training data are prepared from multiple sources. The official training data from subtask 1 \citep{wang2023multimodal} are imported as a json file and converted into a pandas dataframe with text and an emotion for each utterance. Here, data from subtask 1 are used instead of that from subtask 2 because it is likely that the data are identical to each other except that video names are not given for data of subtask 1. Then, the resulting pandas dataframe, translated and converted version of training data from SemEval2024 Task10 \citep{task10dataset} and data by \citet{additionaldata} are imported via CSV file format. As for the two additional datasets, irrelevant columns are dropped so that they are composed of two columns, in which text and emotions are stored respectively. They are concatenated into one dataframe and adjusted to have 7001 utterances (including 7000th counting from 0) for neutral and 5001 at maximum for each one of the other emotions related to this task (surprise, anger, sadness, joy, disgust and fear). 

After this process of data preparation, SpaCy-v3 model \citep{mymodel} is trained using the prepared training data. An unlabeled evaluation dataset is imported as a json file, and the trained model assigns an emotion to each utterance.

At the same step, next sentence prediction with BERT \citep{nextsentenceprediction} assigns an utterance number of emotion cause to each utterance if its assigned emotion is not "neutral". As stated before, it is hypothesised that the emotion cause utterance is either the utterance itself or its previous utterance. Under this assumption, next sentence prediction \citep{nextsentenceprediction} checks whether or not an utterance that it is looking at is related to its previous utterance. The previous utterance is chosen as its emotion cause utterance if these are deemed related, and the utterance itself is chosen if not. After all these processes, lists that include emotions with the utterance numbers and emotion cause utterance numbers (['2\_sadness', '1'] for example) are added to the original evaluation data for submission.

My participation in this task using the combined methodology reveals its limitations of the simple approach to emotion cause analysis. Since my methodology trains SpaCy-v3 model with over 33000 utterances, it is more natural to assume that the simplistic application of next sentence prediction is the main reason for the limited accuracy. My algorithm takes only the utterance itself and its previous utterance into account as possible emotion cause utterances. This premise does not allow my solution to cover cases where one utterance has an influence beyond multiple utterances, limiting the ability to deal with the entire conversation from a macroscopic point of view. On the other hand, there is no doubt that the accuracy of emotion classification is also a reason for the limited ability of my trial. Emotion cause utterances are assigned to non-neutral emotion utterances only, meaning that the algorithm loses its accuracy for both emotion classification and emotion cause analysis at a time if the trained model mistakenly assigns "neutral" to non-neutral emotion utterances.  

% \begin{enumerate}
%     \item \textbf{Official development data translation}: Given the official data that include utterances in Hindi-English code-mixed language and their emotions, translate the utterances into English and save them as a new column.
%     \item \textbf{Additional data concatenation}: Concatenate additional data that have utterances in English and the emotions with the translated official data.
%     \item \textbf{Mitigation of data imbalance}: Separate the concatenated data into each emotion and set a limitation of 3001 utterances (including 3000th counting from 0) per one emotion type.
%     \item \textbf{Model training and prediction}: Train a model with re-concatenated data to predict emotions for unlabeled evaluation data.
% \end{enumerate}







\section{Experimental setup}


\begin{table*}[htbp!]
    \centering
    \setlength{\tabcolsep}{3pt}%
    \begin{tabular}{cccccccccc}\hline
         Dataset& Anger & Disgust & Fear & Joy & Neutral & Sadness & Shame & Surprise & Contempt\\ \hline\hline
         Data from subtask1& 1615 & 414 & 373 & 2301 & 5929 & 1147 & 0 & 1840 & 0\\
         Data from Task10& 819 & 127 & 514 & 1596 & 3909 & 558 & 0 & 441 & 542\\
         Data by Nikam& 4286 & 856 & 5409 & 11037 & 1811 & 6719 & 146 & 4062 & 0\\ \hline
    \end{tabular}
    \caption{Datasets and emotion categories}
    \label{tab:my_label}
\end{table*}


For the emotion classification part, multiple datasets needed to be processed to make an adjusted training dataset.

First of all, the official training data for subtask 1 \citep{wang2023multimodal} are imported as a json file. In the dataset, a conversation ID is assigned to each conversation, and one conversation has multiple utterances. For each utterance, an utterance ID, text, its speaker name and an emotion are assigned. The json file is converted into a pandas dataframe to make the data easier to deal with. Conversation IDs, utterance IDs and speaker names are dropped from the dataframe so that it has only "text"
and "emotion" as columns. Data from SemEval2024 Task10 \citep{task10dataset}, which have Hindi-English code-mixed utterances, are translated into English and converted into a pandas dataframe similarly. Data by \citet{additionaldata} are also imported as a pandas dataframe, and irrelevant columns of the two additional datasets are dropped so that they have text utterances and emotions as columns only. The number of utterances for each different emotion category is as shown on Table1. After these processes, the three dataframes (official training data for subtask 1, data from SemEval2024 Task10 \citep{task10dataset} and data by \citet{additionaldata}) are concatenated into one dataframe in that order. Only first 7001 (including 7000th counting from 0) neutral emotion utterances and  first 5001 utterances for each one of the other emotions are extracted, dropping all the utterances that exceed the limitation from the concatenated dataset to compose an adjusted training data. 

After this data preparation step, SpaCy-v3 model \citep{mymodel} is trained using the adjusted training data, and the trained model is used for emotion classification of the unlabeled evaluation dataset. Next sentence prediction \citep{nextsentenceprediction} is also used for emotion cause analysis as stated in the previous section.







\section{Results}

\begin{table}
    \centering
    \setlength{\tabcolsep}{3pt}%
    \begin{tabular}{cccc}\hline
         w-avg. F1&  F1 & Ranking \\ \hline\hline
         0.1288&0.1389& 12/16 \\ \hline
    \end{tabular}
    \caption{Task scores in evaluation phase}
    \label{tab:my_label}
\end{table}


In the evaluation phase, my solution was tested using the unlabeled test dataset. The result shows a limited ability of my approach, which combines classical machine learning with next sentence prediction under a simplistic assumption. The scores are displayed in Table 2.  




\section{Conclusions}
%What else could we have done
%What worked and didn't work
%We started with the simplest approaches such as SpaCy similarity measures to add to the original baseline measures...





To summarize, my methodology sets a limitation to use textual data only, testing a simple algorithm with a certain premise. I use classical machine learning for emotion classification, and next sentence prediction for identification of emotion cause utterances. 

The next sentence prediction \citep{nextsentenceprediction} in my simple approach takes only the utterance itself and its previous utterance into account, limiting its ability to cover the entire conversation from a macroscopic viewpoint. In addition to that, the accuracy of emotion classification between neutral and non-neutral turned out to be more important than previously thought since it has a significant effect on identification of emotion cause utterances as well.

Although the trial of my simple approach has a large room for improvement, it clearly guides a direction to its future studies. With improvements to enhance the ability to cover conversations from a macroscopic point of view, it might open the door for the potential of my combined methodology.



% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}


\end{document}
