\section{Results}

\label{sec:results}

In this section, we present the experimental findings for both Track A and Track B. To conduct
a comprehensive evaluation, we employed four distinct state-of-the-art sentence embedding models: the powerful general-purpose \textbf{all-mpnet-base-v2}; the lightweight, distilled \textbf{all-MiniLM-L6-v2} optimized for efficiency; the \textbf{LaBSE} (Language-agnostic BERT Sentence Embedding) model specialized for multilingual bitext retrieval; and \textbf{paraphrase-multilingual-mpnet-base-v2}, a multilingual variant trained on paraphrase data. For brevity, we use \textbf{para-mul-mpnet-v2} to denote the \textit{paraphrase-multilingual-mpnet-base-v2} model in subsequent tables and analysis.

\subsection{Track A Results}

In Track A, our system achieved a private score of \textbf{0.6925}, securing the \textbf{14\textsuperscript{th}} position out of 42 participating teams. For reference, the top-performing system on the leaderboard reached 0.7800. As shown in Tables~\ref{tab:track_a_dev} and Tables~\ref{tab:track_a_test}, pseudonymized data yields consistent improvements. The full Dev-set comparison is provided in Appendix~\ref{appendix:experimental-setup}.

% \paragraph{Manual Weight Tuning.}
% The fusion weights are selected through manual tuning on the development set by comparing a small set of interpretable configurations in Table~\ref{tab:track_b_fusion_ablation}. We retain the submitted setting $(w_{\text{full}}, w_{\text{theme}}, w_{\text{plot}}, w_{\text{outcome}}) = (0.50, 0.10, 0.20, 0.20)$ because it yields the highest Dev score (0.7250). 

\begin{table}[!ht]
\centering
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{5pt}
\begin{tabular}{lcc}
\hline
\textbf{Model} & \textbf{Raw Data} & \textbf{Pseudonymized Data} \\ \hline
all-mpnet-base-v2 & 0.6650 & \textbf{0.6925} \\
para-mul-mpnet-v2 & 0.6625 & 0.6300 \\
LaBSE & 0.6050 & 0.5975 \\
all-MiniLM-L6-v2 & 0.6125 & 0.5800 \\ \hline
\end{tabular}
\caption{Performance comparison of transformer models on Track A (Test set). Bold indicates the best overall score.}
\label{tab:track_a_test}
\end{table}

\paragraph{Performance Analysis.}
The results demonstrate that \textbf{all-mpnet-base-v2} consistently outperforms other models, achieving the highest scores on both Dev and Test sets with pseudonymized data. This aligns with benchmarks on the Massive Text Embedding Benchmark (MTEB) leaderboard, where MPNet's Masked and Permuted Language Modeling (MPLM) objective allows it to capture dependency information more effectively than standard BERT-based models~\cite{song2020mpnet, muennighoff2022mteb}.

\paragraph{Impact of Pseudonymization.}
Pseudonymization yields consistent improvements for the top-performing MPNet model (increasing from 0.6650 to 0.6925 on the Test set). We hypothesize that pseudonymization acts as a normalization step, removing entity-specific noise and forcing the model to focus on the semantic structure of fictional narratives rather than overfitting to specific proper nouns.

\subsection{Track B Results}
For Track B, our team ranked \textbf{6\textsuperscript{th}} out of 25 participants, achieving a private score of \textbf{0.6875} on the Codabench platform (where the top score was 0.7200). Our methodology involved testing various Large Language Models (LLMs): gpt-4o-mini, gpt-4.1, and o3-mini to extract structured schemas from the narratives.
Detailed Dev-set results for Track B are reported in Appendix~\ref{appendix:experimental-setup}.

% \paragraph{Dev--Test Generalization Gap.}
% Track B exhibits a noticeable dev-to-test discrepancy. The best Dev configuration reaches 0.7250 (\texttt{gpt-4.1} + \texttt{all-mpnet-base-v2}), whereas the best Test score is 0.6875 (\texttt{o3-mini} + \texttt{all-mpnet-base-v2}). This gap suggests limited transfer from the development distribution to the private test setting and indicates sensitivity to the choice of LLM extractor. One plausible explanation is that extractor-specific schema style (e.g., event granularity, phrasing consistency, and abstraction level) affects downstream embedding quality differently across splits. These results motivate stronger regularization and extractor-robust training strategies in future work.

\begin{table}[!ht]
\centering
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{4pt}

\begin{tabularx}{\linewidth}{Xccccc}
\hline
\textbf{Configuration} & $\mathbf{w}_{\text{full}}$ & $\mathbf{w}_{\text{theme}}$ & $\mathbf{w}_{\text{plot}}$ & $\mathbf{w}_{\text{outcome}}$ & \textbf{Dev Score} \\ \hline
Equal weights & 0.25 & 0.25 & 0.25 & 0.25 & 0.625 \\
Full-only & 1.00 & 0.00 & 0.00 & 0.00 & 0.65 \\
Submitted (dev-tuned) & 0.50 & 0.10 & 0.20 & 0.20 & \textbf{0.725} \\
Higher theme & 0.40 & 0.30 & 0.15 & 0.15 & 0.675 \\ \hline
\end{tabularx}

\caption{Fusion-weight ablation on Track B (Dev set). Bold indicates the best overall score.}
\label{tab:track_b_fusion_ablation}
\end{table}

% \paragraph{Manual Weight Tuning.}
% The fusion weights are selected through manual tuning on the development set by comparing a small set of interpretable configurations in Table~\ref{tab:track_b_fusion_ablation}. We retain the submitted setting $(w_{\text{full}}, w_{\text{theme}}, w_{\text{plot}}, w_{\text{outcome}}) = (0.50, 0.10, 0.20, 0.20)$ because it yields the highest Dev score (0.7250), indicating that emphasizing the full-text embedding while preserving moderate plot/outcome signals provides the best balance for this setup.

\begin{table}[!ht]
\centering
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{5pt}
\begin{tabular}{llc}
\hline
\textbf{Embedding Model} & \textbf{LLM Extractor} & \textbf{Score} \\ \hline
all-mpnet-base-v2 & o3-mini & \textbf{0.6875} \\
all-mpnet-base-v2 & gpt-4.1 & 0.6391 \\
all-mpnet-base-v2 & gpt-4o-mini & 0.6291 \\
LaBSE & o3-mini & 0.6291 \\
LaBSE & gpt-4.1 & 0.6466 \\
LaBSE & gpt-4o-mini & 0.6541 \\
para-mul-mpnet-v2 & o3-mini & 0.5915 \\
para-mul-mpnet-v2 & gpt-4.1 & 0.5940 \\
para-mul-mpnet-v2 & gpt-4o-mini & 0.5614 \\
all-MiniLM-L6-v2 & o3-mini & 0.5414 \\
all-MiniLM-L6-v2 & gpt-4.1 & 0.5664 \\
all-MiniLM-L6-v2 & gpt-4o-mini & 0.5338 \\ \hline
\end{tabular}
\caption{Track B results across different LLM-based schema extractors and embedding models on Track B (Private test set). Bold indicates the best overall score.}
\label{tab:track_b}
\end{table}