% Details here
% https://semeval.github.io/paper-requirements.html
\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[final]{acl}
\usepackage{multirow, makecell}
\usepackage{amsmath}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{amssymb}
\usepackage{enumitem}
\setlist{nosep, leftmargin=*}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{adjustbox}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{graphicx}
\usepackage{subcaption}
\newcommand\todo[1]{\textcolor{red}{#1}}
\usepackage{xcolor,colortbl}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{tcolorbox}
\tcbuselibrary{breakable}
\usepackage{mwe}
\usepackage{tabu}

\usepackage{microtype}

% Compact spacing for professional layout
\setlength{\parskip}{0.3ex plus 0.1ex minus 0.1ex}

\input{cmds}


\title{hdharpure at SemEval-2026 Task 3: BERT-Based Modeling and Prediction Behavior Analysis for Multilingual Valence--Arousal Scoring}

\author{
  Harshal Dharpure\textsuperscript{1}, Nicolay Rusnachenko\textsuperscript{2}\\
  \textsuperscript{1} Indian Institute of Technology, Patna \\
  \textsuperscript{2} Centre for Applied Creative Technologies (CFACT+), Bournemouth University, UK  \\
  \texttt{harshal\_2511ai30@iitp.ac.in}, 
  \texttt{nrusnachenko@bournemouth.ac.uk}
} 



\date{January 2024}

\begin{document}

\maketitle


\begin{abstract}
The SemEval-2026 Task 3 is a 
\textit{Dimensional aspect-based sentiment analysis} (DimABSA) task which extends traditional ABSA by predicting continuous regression in two dimensions: valence (V) and arousal (A). 
% Task.
The Track~A/Subtask~1 represent multilingual task in which for a given text and aspects mentioned in it, there is a need to predict V/A scores for each aspect. 
% Methodology.
Our approach is based on the pretraining-finetuning concept: we first pretrain multilingual model ($M'$) followed by its fine-tuning ($M''_{l,d}$) on the training data of specific domain ($d$) and language ($l$).
% Contribution.
We adopt \roberta{} (\model{}) as the encoder with separate regression heads for valence and arousal prediction. 
% Solution.
Our experiments on manual split of official SemEval-2026 Task 3 dataset (\dimabsaTrainEval{}) demonstrate that fine-tuning model in two stages ($M''_{l,d}$) results in average \improvement{} times improvement by $\rmse{}$ over approach of direct fine-tuning ($M_{l,d}$). 
% Result analysis.
To investigate limitations of the existing approach we visualize and discuss limitations of our system.
% Code.
Our code is publicly available\footnote{\url{https://github.com/harshalDharpure/SemEval-2026-Task-3}}.

\end{abstract}

\section{Introduction}
\label{sec:introduction}

Aspect-based sentiment analysis (ABSA) is a task of predicting sentiment for specific aspects mentioned in text~\cite{liu2012sentiment}. 
\textit{Dimensional} Aspect-Based Sentiment Analysis (DimABSA)~\cite{yu-etal-2026-semeval,lee2026dimabsa} is the task introduced at SemEval-2026 as an extension of traditional ABSA with the prediction of continuous scores~\cite{russell1980}. The scores correspond to the following parameters: \textit{valence} and \textit{arousal}. 

\begin{figure*}[!t]
 \centering
  \includegraphics[width=\textwidth]{pics/pipeline.drawio.png}
  \caption{Pretraining-finetuning concept adaptation for model $M$ (left), architecture of the model $M$ (right)}
  \label{fig:pipeline}
\end{figure*}

The Transformer architecture~\cite{vaswani2017attention} caused a significant impact on the field of Natural Language Processing (NLP), including field of Sentiment analysis. 
Encoder-based language models features a robust framework for establishing the connection between texts and labels~\cite{rogers-etal-2020-primer}.
BERT and its derivative variants~\cite{devlin-etal-2019-bert, liu2019robertarobustlyoptimizedbert} has become widely distributed in various tasks that required classification output.

The DimABSA dataset (\dimabsaDataset{}) and its training part particularly (\dimabsaTrainData{}) shares a large amount of annotated data that unites texts in different languages: 
English~(\texttt{eng}), 
Japanese~(\texttt{jpn}), 
Russian~(\texttt{rus}), 
Ukrainian~(\texttt{ukr}), 
Tatar~(\texttt{tat}), 
Chinese~(\texttt{zho}). 
For each language, the data from the following set of domains presented:
restaurants, laptop, finance. 
To maximise the use of cross-lingual and cross-domain data, in this paper we employ \textit{pretraining-finetuning}~\cite{wang2024tutorialpretrainfinetuneparadigmnatural} technique to adapt encoder-based transformers for V/A scoring. We choose \roberta{}~\cite{liu2019robertarobustlyoptimizedbert} for experiments as it is one of the most widely used multilingual transformers.
According to our experiments 
on manually arranged split of training data (\dimabsaTrainEval{}), 
we demonstrate that computing pre-training techniques with further fine-tuning result in \improvement{} improvement.
To study prediction behaviour of the result systems by visualizing alignments in two directions:~(i)~discrepancies of predicted results from \textit{etalon} (ground-truth) annotations and (ii)~discrepancies of etalon annotations from data distribution in predictions. 
By relying on the observed results on \dimabsaTrainEval{}, we discuss limitations of our approach and suggest further directions to address them.

\section{Methodology}
\label{sec:methodology}

Given $C_{l,d}$ is a collection of texts 
for the particular language $l \in \languages{}$ and domain $d \in \domains{}$. 

\textbf{Task description.} 
For the particular text $t \in C_{l,d}$ with set of aspects mentioned in it $\{\alpha_1, \alpha_2, ..., \alpha_n\}$, 
for each aspect $\alpha_i$, there is a need to predict V/A pair ($v_i', a_i'$) scores.
The $v_i'$ and $a_i'$ are in the range $[1.0, 9.0]$.

Figure~\ref{fig:pipeline}~(left) illustrates the pretraining and fine-tuning concept adaptation for model $M$.
Given model $M$ we compose its domain specific version ($M''_{l_i,d_j}$) by following steps: 
\begin{enumerate}
    \item \textbf{Pre-training}: we unite $C_{l,d}$ across all the languages $l \in \languages{}$ and domains $d \in \domains{}$ into single collection $C$ to fine-tune $M$, with result model denoted as $M'$.
    \item \textbf{Fine-tuning}: for each and particular language $l_i \in \languages{}$ and domain $d_j \in \domains{}$ we use $M'$ to prepare domain specific version ($M''_{l_i,d_j}$).
\end{enumerate}

In our methodology, $M$ is a machine learning model that for a given input context outputs valence / arousal pair scores, with values cropped in accordance with task requirement.
We use Mean Squared Error (MSE) loss for both valence and arousal prediction and compose \textsc{combined loss} (see Figure~\ref{fig:pipeline}) as follows:
\begin{equation}
  \nonumber
  \mathcal{L} = \text{MSE}(\hat{v}, v) + \text{MSE}(\hat{a}, a)
\end{equation}
where $v$ and $a$ are ground truth valence and arousal scores, respectively.

Figure~\ref{fig:pipeline}~(right) illustrates the architecture of the model $M$.
We use BERT-based transformer~\cite{liu2019robertarobustlyoptimizedbert}, which supports input for two sentences separated by meta-token (\texttt{[SEP]}) and prefixed by token of class (\texttt{[CLS]}).
For the given text $t$ and particular aspect mentioned in it ($\alpha$) we compose input context as follows:
\begin{align}
    \nonumber
    \texttt{[CLS]}~\alpha~\texttt{[SEP]}~t~\texttt{[SEP]}
\end{align}

To obtain V/A scores we use mean pooling over the input context token sequence, weighted by the attention mask to ignore padding.
The pooled vector is passed through a fully connected layer (FC) with LayerNorm and dropout:
\begin{align}
    \nonumber
    h_{t} &= \text{LayerNorm}(\text{FC}(h_{pooled}))
\end{align}
and then through two heads:
\begin{align}
    \nonumber
    \hat{v} &= W_v \cdot h_{t} + b_v
    \\
    \nonumber
    \hat{a} &= W_a \cdot h_{t} + b_a
\end{align}
where $W_v$ and $W_a$ are weight matrices for valence and arousal respectively;
$\hat{v}$ and $\hat{a}$ are predicted valence and arousal scores, $\hat{v}, \hat{a} \in \mathbb{R}$. To meet the requirement of the task, valence and arousal scores additionally cropped to the range $[1.0, 9.0]$ as follows:
\begin{align}
    \nonumber
    v = \max(1.0, \min(\hat{v}, 9.0))
    \\
    \nonumber
    a = \max(1.0, \min(\hat{a}, 9.0))
\end{align}

\input{tables/results.tex}


\input{tables/dataset_stats.tex}

\section{Dataset}
\label{sec:dataset}

The official dataset (DimABSA)~\cite{lee2026dimabsabuildingmultilingualmultidomain,yu-etal-2026-semeval} and available amount of sentences in its training part (\dimabsaTrainData{})  presented in Table~\ref{tab:dataset_stats}.
In this paper we construct splits of the \dimabsaTrainData{} that correspond to: 
\begin{itemize}
    \item \dimabsaTrainActual{} - dataset for pre-training and fine-tuning;
    \item \dimabsaTrainEval{} - dataset for non-official experiments.
\end{itemize}
The splitting is performed at the sentence level with inclusion of corresponding aspects mentioned in it. We apply a random 80/20 split to construct \dimabsaTrainActual{} and \dimabsaTrainEval{}; we keep all aspects of the same sentence together.

\subsection{Experimental Setup}

The full model (encoder and regression heads) is trained for 5 epochs with learning rate $1.5 \times 10^{-5}$ with 16 bit float precision. 
For the pre-training we use 5 epochs.
At the Fine-tuning trained with 3 epochs. 
We use \robertaBase{} as the encoder~\cite{liu2019robertarobustlyoptimizedbert} and its initial state further in this section refered as $M$.

\input{tables/params.tex}

For the described methodology in Section~\ref{sec:methodology} we set up the following experiments that involve application of the original model $M$ and its pre-trained version $M'$ pre-trained on the \dimabsaTrainActual{} (see Section~\ref{sec:dataset}).
For the given language $l_i$ and domain $d_j$ we set up the following experiments:
\begin{itemize}
    \item \textbf{Exp1}: Direct fine-tuning $M$ on target language/domain (no pretraining)
    \item \textbf{Exp2}: Inference only $M'$ (no fine-tuning)
    \item \textbf{Exp3}: Fine-tuning $M'$ to obtain $M''_{l_i,d_j}$ used to infer results. 
\end{itemize}

\input{figure_analysis.tex}

For this task, organizers provided the following evaluation metric: RMSE$_{VA}$, which is defined as:
{\small
\begin{equation*}
    \rmse{} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \left[(v_{i}^{'} - v_{i})^2 + (a_{i}^{'} - a_{i})^2\right]}
\end{equation*}
}%
where $n$ corresponds to a total number of pairs $\left<t, \alpha\right>$, and 
$v_{i}$ and $a_{i}$ are ground truth valence and arousal scores, respectively. Lower values indicate better performance.


\section{Result Analysis and Discussion}
\label{sec:result-analysis-and-discussion}

Table~\ref{tab:results} provides results by $\rmse{}$ for the experiments (Exp1, Exp2, Exp3), separately for \dimabsaTrainEval{} and \dimabsaTest{} and each language and domain individually. 
%\textcolor{red}{We note that \dimabsaTrainEval{} is a random hold-out from the training distribution, while \dimabsaTest{} is an unseen official split; therefore, a substantial RMSE gap can occur due to distribution shift (topic/style, aspect density) and label-space sparsity/outliers discussed below.}

According to the results observed on \dimabsaTrainEval{} for the \robertaBase{} ($M$), using adapting pretraining-finetuning (Exp3) results in \improvement{} times improvement\footnote{ratio of results obtained in Exp1 to one in Exp3} 
by $\rmse{}$
over results from approach Exp~1.
%\textcolor{red}{This improvement reflects the effect of the pretraining stage ($M \to M'$) followed by language/domain fine-tuning ($M' \to M''_{l_i,d_j}$) as defined in Section~\ref{sec:methodology}.}

Our final submission for evaluation on the official split (\dimabsaTest{}) corresponds to the results of the $M''_{l_i,d_j}$ models (Exp3).
For the comparison reason, we include results of baseline approaches shared by DimABSA organizers, obtained via \kimiBaseline{}\footnote{\url{https://huggingface.co/moonshotai/Kimi-K2-Thinking}} and \qwenBaseline{}\footnote{\url{https://huggingface.co/Qwen/Qwen3-14B}}.
We found that results of our final submission strongly outperform results of \qwenBaseline{} and
\kimiBaseline{} in \qwenImprovement{} and \kimiImprovement{} times by $\rmse{}$ respectively. 
The exception reprsent the subset of \dimabsaTest{} for Tatar language (Restaurant domain), in which \kimiBaseline{} demonstate slightly superior performance to our approach ($\approx1.06$ times by $\rmse{}$).

\textbf{Result Analysis.}
For the predicted scores of the particular model (pred) and related etalon scores (etalon),
we investigate the behavior by visualizing alignment of:
\begin{enumerate}
\item \textit{pred$\to$etalon} --  alignment of scores obtained from $M''_{l_i,d_j}$ towards etalon scores; this allow us to locate source of errors etalon dataset;
\item \textit{etalon$\to$pred} --  alignment of etalon scores towards scores classified by $M''_{l_i,d_j}$;
\end{enumerate}

To make comparison of the related analysis consistent across different languages, 
we choose `restaurants' domain as the most common domain (See Table~\ref{tab:dataset_stats}).

Figure~\ref{fig:analysis} presents the related analysis for $M''_{l_i}$ models (Exp3) on restaurants domain from \dimabsaTrainEval{} dataset.
Given scores obtained from $M''_{l_i}$ ($pred$), and ground-truth scores ($etalon$), we analyse the alignment of \textit{pred} and \textit{eval} in two directions:~
$pred\to etalon$, and 
$etalon \to pred$.
According to the related analysis, we identified the following problems that impact the performance:
\begin{enumerate}
    \item \textbf{cropped etalon space}: the etalon space has no information for arousal ($a$) scores in range $a \in [0,2]$ (pred$\to$etalon block in Figure~\ref{fig:analysis});
    \item \textbf{presence of outliers}: examples valence/arousal scores of $\left<5.0,9.0\right>$ across various languages (rus, tat, ukr) (etalon$\to$pred block in Figure~\ref{fig:analysis});
    \item \textbf{formation of error clusters}; example: areas $v \in \left<0,3\right>, a \in \left<6,9\right>$ has relatively high RMSE (etalon$\to$pred block in Figure~\ref{fig:analysis});
\end{enumerate}

To address these problems, we believe that the research around proper exploit of \texttt{[CLS]} token output of BERT-based is important. 
As a one approach, the Natural Language Inference (NLI) technique~\cite{sun2019utilizing} to~(i)~supply prompt hint after separator token (\texttt{[SEP]}), and
(ii)~exploit output from \texttt{[CLS]} token in area classification and fact-checking.
To provide numerical definitions with their lexical definitions, one approach we provisionally see is to re-center V/A space by defining larger areas to be referred as high / middle / low subareas of related V/A scores.


\section{Conclusion}
\label{sec:conclusion}

This paper presents a study on adapting encoder-based transformers in a novel task, dubbed as dimensional aspect-based sentiment analysis (DimABSA) presented at SemEval-2026 as Task~3.
This task extends the traditional ABSA with the prediction of continuous scores for valence and arousal.
DimABSA has a multilingual and multidomain setup, and to leverage the use of cross-lingual and cross-domain data, we this paper contribute by investigating effect of pre-training on the model performance.
According to our experiments that involve application of \robertaBase{} model, we demonstrate that pretraining-finetuning concept results in \improvement{} improvement by $\rmse{}$ over direct domain and language specific fine-tuning.

\bibliography{main}

\end{document}