Our aim is to investigate if, across years, languages and domains, the target terms \textit{feminist} and \textit{feminists} undergo a connotative shift. In order to do that, we employ two methods: \textbf{frame analysis} \cite{Entman1993Framing} and \textbf{classification}.
The frame analysis has been used to identify how certain aspects in an issue’s portrayal are emphasized and to characterize societal attitudes in the annotated dataset. This is crucial for capturing the connotations of ``feminists'' which can vary significantly depending on the context. For instance, the term may be used with positive connotations in discussions of gender equality, but with negative connotations in prejudiced or hostile comments.
By leveraging frame analysis, we aim to capture these nuanced connotations and associations, thereby gaining a more comprehensive understanding of the term's usage. 
%In order to characterize societal attitudes at scale, we employ classification tools, including both \textbf{encoder-based models} and \textbf{Large Language Models (LLMs)}.

%CWE, unlike traditional static embeddings, consider the surrounding words when generating a word's representation. This is crucial for capturing the connotations of ``feminists'' which can vary significantly depending on the context. For instance, the term may be used with positive connotations in discussions of gender equality but with negative connotations in prejudiced or hostile comments. By leveraging CWE, we aim to account for these subtle semantic connotations and associations, and gain a more comprehensive understanding of the term's usage.

While frame analysis offers interpretability and language-agnostic robustness, it is inherently limited by the coverage of the annotations. %These methods capture general semantic orientation but often fail to model subtle contextual nuances, emergent meanings, or implicit associations—especially in noisy, user-generated content.
To complement this, we employ \textbf{encoder-based models} and \textbf{Large Language Models (LLMs)}.
We fine-tune encoder-based models on annotated data. We employ Twitter-XLM for English, \cite{barbieri-etal-2022-xlm}, AI-Sweden RoBERTa\footnote{\url{https://huggingface.co/AI-Sweden-Models/roberta-large-1160k}} for Swedish, and AlBERTo for Italian \cite{PolignanoEtAlCLIC2019}. All models were trained on Twitter data, with the exception of the Swedish model, which was trained on a dataset that also includes Reddit content.
For Large Language Models, we use GPT-4o mini in zero-shot and few-shot settings, where one instance for each frame is reported in the prompt.

%\subsection{Connotation Analysis}
%The core methodology of this work is comparing CWE of \textit{feminist} and \textit{feminists} to anchor words for negative and non-negative associations, which we call seed words. Seed words represent frames associated with feminists. For instance, the seed word \textit{misandric} represents all posts where feminists are framed to hate men. Therefore, seed words are not necessarily co-occurrences of \textit{feminist(s)}.
%The complete list of seed words and how they are retrieved can be found in \ref{sec:exp}. 

%As a preliminary step to evaluate the methodology proposed, a set of five Italian 

%If a sentence contains two or more instance of that target words, we average the embeddings.
%From a manual inspection, xl-lexeme and GPT produced the embeddings to be most aligned with human judgments. Therefore, we proceed with xl-lexeme and GPT, which are suitable for all target languages.

\begin{comment}
    
\subsection{Semantic Axis Method}
The Semantic Axis Method \cite{jurafsky2023speech} operates by utilizing the seed words to determine the sentiment alignment of a target word \textit{w}, in our case \textit{feminist, feminists} in English, Italian, Spanish and Swedish. 

%The semantic similarity between two words is approximated by the cosine similarity between their vectors. 
%At the end, we average all the embeddings created for the target terms each year, in order to have an embedding that represents a whole year.

Given a target word \( w \), its embedding \( \mathbf{v}_w \), and the semantic axis \( \mathbf{a} \), we compute the cosine similarity:

\[
\text{Score}(w) = \cos(\theta) = \frac{\mathbf{v}_w \cdot \mathbf{a}}{\|\mathbf{v}_w\| \|\mathbf{a}\|}
\]
If the score is close to 1, the target word is highly associated with one side of the axis (V$+$), while a negative score means alignment with the other (V-).
Given the average embeddings of negative words (\(\mathbf{e}_{\text{neg}}^{\text{avg}}\)) and non-negative words (\(\mathbf{e}_{\text{non-neg}}^{\text{avg}}\)), the feminist axis is computed as:

\[
\mathbf{a}_{\text{feminist}} = \mathbf{e}_{\text{non-neg}}^{\text{avg}} - \mathbf{e}_{\text{neg}}^{\text{avg}}
\]

where:

\[
\mathbf{e}_{\text{neg}}^{\text{avg}} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{e}_{\text{neg}, i}, \quad
\mathbf{e}_{\text{non-neg}}^{\text{avg}} = \frac{1}{M} \sum_{j=1}^{M} \mathbf{e}_{\text{non-neg}, j}
\]

with \( N \) and \( M \) being the number of negative and non-negative words, respectively.
We report the score of similarity between target words and the semantic axis for each year, language and domain.


%Additionally, we perform K-Means clustering on the contextual embeddings to identify patterns in their semantic distribution that could indicate if a certain topic or event causes a connotative shift.


%\paragraph{Validation}
%We base the evaluation of our approach on two assessments: qualitative analysis and comparison with LlaMa3 predictions on annotated data. 
%For the qualitative analysis we make use of the interactive plot implementation of Plotly which allows to manually check the instances positioned in the negative and in the positive side of the axis, as an evidence that our approach captures diverse semantic connotations with respect to the target terms \textit{feminist(s)}.
%Results of the manual inspection will be presented in Section \ref{sec:exp}.
%For the comparison with LLaMa, we ask the model to predict whether \textit{feminist, feminists} have a negative connotation in the input sentences. LlaMa obtains an average F1 score of 0.57 on 2k annotated sentences in Italian.

\end{comment}

\begin{comment}
\subsection{Diachronic Embeddings}
The goal of diachronic word embedding model is to automatically discover the changes in the usage of a word.
The current usage at time t of a word w can be obtained by inspecting the top nearest words of w.
To achieve this, we leverage the Living With Machine library to train Diachronic Contextualized Word Embeddings on each year and each domain. Then, we extract the top 20 nearest neighbors to our target terms.
In this way we can observe which word vectors in the time period t are the closest to the vector of our target words.
We train separate word embedding models for each of the time intervals, which represent the year, and then vectors from these models are aligned in a common space using orthagonal procrustering.
\end{comment}

%\subsection{LLM's Bias}
%\todo{IN PROGRESS}
%We prompt a variety of LLMs in a zero-shot setting to complete the sentences \textit{feminists are...} and \textit{people think feminists are...} and the translated version in Spanish, Italian and Swedish.
%We want to investigate if LLMs are sensitive to cultural stereotypes about feminists.
%We prompt the model to provide us 10,000 answers and we compute the probability of each word.
%The sentence "For an american feminist she's not even such a butterball" from our dataset implies the stereotype that feminists are fat. 
%We experiment with the following models: LlaMa2, LlaMa3, Mistral v-3, Quen. We set the temperature to 0.7 for all of them.
%its capacity in finding a stance toward feminists. 
%\begin{figure}
 %   \centering
  %  \includegraphics[width=1\linewidth]{download-21.png}
   % \caption{“Feminists are" with LLama3 in Italian.}
   % \label{fig:download_21}
%\end{figure}
%\begin{figure}
 %   \centering
  %  \includegraphics[width=1\linewidth]{download-22.png}
   % \caption{“People think that feminists are" with LLama3 in Italian.}
    %\label{fig:download_22}
%\end{figure}
%When the model is asked to complete the sentence feminists are, it refuses to provide negative stereotypes. However, when it is asked what people think, the model tends to give negative answers, which are not linguistically correct. The model tends make up adjectives whose meaning can be intuited, but do not exist. This is the case for \textit{fieroveggenti, lottizate, antimascisti}.
%\todo{Amanda, Emi, Flor: can you translate these 2 sentences?}

\begin{comment}
\subsubsection{Stance Detection with LLMs}
\todo{TO DO maybe for next paper?}
We compare two approaches:
\begin{itemize}
    \item prompting the LLM only with the task, relying on its implicit knowledge;
    \item providing the task and the guidelines used for annotation.
    \item providing the task, the guidelines used for annotation and ask to report the number of the guideline used for the decision. This aids explainability.
\end{itemize}

\subsubsection{Chain-of-Thought}
\todo{TO DO}
Moreover, we select all misclassified instances and we ask the model to reason step-by-step on why a certain stance was assigned to that example. In this way we can evaluate the model's bias towards feminists.
\end{comment}
\begin{comment}
\section{Pipeline}
\begin{enumerate}
    \item Create vectors for contexualized embeddings for annotated sentences and through semantic similarity map unlabeled sentences --> implicit annotations. Check some examples to see if it makes sense (e.g. 100 for language?). Create X vector spaces for X years (to analyze the negative connotation over time); Create 2 vector spaces to compare in-domain with out-domain connotations.
    \item Compare vectors with seed words, divided between positive (activist, thoughtful...) and negative (all the negative stereotypes we found from corpus analysis or through LLMs) --> SAM method.
    %\item optionally: aspect-based sentiment analysis because a sentence could be neutral but contain positive and negative aspects about feminists and then split the sentences based on that to create separate embeddings for each part of the sentence (maybe it's too much and can be left for future work)
    \item Analysis with LLMs: prompt few-shots llms to find out if they show biases (simply asking if the sentence has a negative connotation or not). We could also ask for the stance --> target-based stance. See \ref{sec:prompts} for prompts.
    \item CoT --> we take misclassified instances and we ask the model to reason step by step. Why did you say that this tweet has a certain stance?
    \item topic modelling on all tweets
    \item compare connotation between feminist and 'woman' -- feminist maybe it's negative also out-domain while 'woman' should be more neutral.
    \item measure intensity of hate between feminist and woman to show that hate vs feminist is more violent than vs women
\end{enumerate}

If we don't want to annotate data in EN and ES we could try zero-shot cross-lingual experiments with the semantic similarity: 
we use IT annotations to create data points and we compare the other languages by computing semantic similarity either by translating instances or using mBERT (or both).

\subsection{Tools}
\begin{enumerate}
    \item https://github.com/pierluigic/xl-lexeme for creating WordinContext vectors
    \item https://www.changeiskey.org/event/2024-esslli/ for creating contextualized embeddings
\end{enumerate}
\subsection{Metrics for Evaluation}
\begin{description}
\item[Human Evaluation] for contextualized embeddings: we need to check that instances clustered together have negative vs neutral connotation of feminist/s. 
To evaluate our method, we observe whether detected semantic shifts are positively correlated with
human judgements.
    \item[F1 SCORE] for binary classification (negative perception vs not).
    \item[Character overlap] for stereotypes.
\end{description} 

\end{comment}