
\section{Experiments}\label{experiments}
We have conducted our experiments utilizing the dataset provided by the task's organizers that is explained in Section \ref{dataset}. For the models based on prompting, we utilized only the test and practice\_test datasets, whereas the training data was employed exclusively for fine-tuning the classification-based models. Beyond experimenting with models within our ensemble framework, we also explored the integration of results from fine-tuned classification models as a form of external knowledge within the context of prompting. The efficacy of all models is evaluated using three metrics: Macro F1 Score, Faithfulness, and Consistency, each of which is briefly described in Section \ref{evaluation}.

\subsection{Dataset}\label{dataset}

The corpus presented for analysis encompasses training, development, practice\_test, and test datasets, each containing a distinct number of samples. Table \ref{tab:data} displays the quantity of samples for each dataset. The content of each sample, including statements and evidence, has been reconstructed by a collaborative effort of clinical domain experts, clinical trial organizers, and research oncologists associated with the Cancer Research UK Manchester Institute and the Digital Experimental Cancer Medicine Team.

% Please add the following required packages to your document preamble:
% \usepackage{graphicx}
\begin{table}[ht!]
\centering
    
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Split} & \textbf{\#Samples} & \textbf{\#Entailment} & \textbf{\#Contradiction} \\ \hline
Train          & 1700               & 850                   & 850                      \\ \hline
Practice\_test & 2142               & 730                   & 1412                     \\ \hline
Development    & 200                & 100                   & 100                      \\ \hline
Test           & 5500               & 1841                  & 3659                     \\ \hline
\end{tabular}%
}
\caption{Overview of Dataset Splits: Distribution of Samples, Entailment, and Contradiction Labels}
    \label{tab:data}
\end{table}



\subsection{Evaluation}\label{evaluation}
In assessing system performance, the organizers, in conjunction with the macro F1 score, opted to examine model efficacy on a contrast dataset comprising statements with interventions. The comprehensive ranking of the systems is determined by the mean of two novel metrics: Faithfulness (as defined in Equation. \ref{faith}) and Consistency (as defined in Equation. \ref{cons}), across all types of interventions.

\begin{equation}\label{faith}
   FaithFulness = \frac{1}{N}\sum_{1}^N |f(y_i) - f(x_i)|   
\end{equation}
where \(x_i \in C : Label(x_i) \neq Label(y_i),  f(y_i) = Label(y_i)\).
\begin{equation}\label{cons}
    Consistency = \frac{1}{N}\sum_{1}^N 1 - |f(y_i) - f(x_i)|
\end{equation}
where \(x_i \in C : Label(x_i) = Label(y_i)\). Faithfulness quantifies the degree to which a system reaches an accurate prediction based on the correct rationale. While, Consistency measures the degree to which a system yields identical outputs for semantically equivalent queries. The results obtained during the experimental trials are presented in the subsequent section. 

