The data used for this task is comprised of 1,000 breast cancer CTRs collected from ClinicalTrials.gov with 24,000 entailment relations annotated by clinical domain experts \cite{jullien-etal-2023-nli4ct}. Each CTR is comprised of 4 sections: eligibility criteria, intervention, results and adverse events.

% \noindent \textbf{Eligibility criteria:} A set of requirements for the inclusion or exclusion of patients in a clinical trial.

% \noindent \textbf{Intervention:} Information on the type of treatment, its dosage, frequency, and duration.

% \noindent \textbf{Results:} Outcome measures, units, and the results of the clinical trial.

% \noindent \textbf{Adverse Events:} Signs and symptoms experienced by the participants during the clinical trial.

Each sample from the NLI4CT dataset consists of the CTR premise (one of the 4 sections of the CTR), a statement, and an entailment label (Entailment or Contradiction). The premise can refer to only one CTR in the \textit{Single} type instance, or to a primary and a secondary trial in the \textit{Comparison} type. 
%As opposed to last year's shared task \cite{jullien-etal-2023-semeval} in which participants had to predict the entailment relation and retrieve evidence for the prediction,
The purpose of the current task is to evaluate the consistency of models and their ability to perform faithful reasoning \cite{jullien-etal-2024-semeval}. For this purpose, different semantic altering 
(Contradiction Rephrasing and Numerical Contradiction) 
or semantic preserving interventions 
(Paraphrase, Numerical Paraphrase and Definition) 
have been conducted on the evaluation data. The NLI4CT dataset consists of 1,700 entailment relations in the training split, 200 in dev, and 5,500 in the test split.

% These interventions are detailed below:

% \noindent \textbf{Paraphrase} (e.g., ``Nobody taking part in the primary trial suffered a UTI" -> ``there were no occurrences of urinary tract infections among the participants of the primary clinical trial")

% \noindent \textbf{Contradiction Rephrasing} (e.g., ``There was at least 1 case of infection in both the primary trial and the secondary trial" -> ``there were no reported infections in the primary clinical trial nor the secondary clinical trial")

% \noindent \textbf{Numerical Paraphrase} (e.g., ``365 days"  -> ``52 weeks")

% \noindent \textbf{Numerical Contradiction} (e.g.,  ``3 years" -> ``8 years")

% \noindent \textbf{Definition} appended to the original statement (e.g., “gingival hemangioma is a hemangioma arising from the gums.”)

% (Figure \ref{tab:dataset}).

% \begin{table}[hbt!]
%     \centering
%     \begin{adjustbox}{width=0.35\textwidth}
%     \begin{tabular}{ccc}
%     \toprule
%      \textbf{Split}&  \textbf{Intervention}& \textbf{Samples}\\
%      \midrule
%      Train&  -& 1,700\\
%      \specialrule{.01em}{0em}{0em}
%      Dev&  -& 200\\
%      \specialrule{.01em}{0em}{0em}
%      \multirow{ 6}{*}{Test}&  -& 500\\
%      &  Paraphrase& 1,500\\
%      & Contradiction&1,500\\
%      & Definition&1,500\\
%      & Numerical Paraphrase&224\\
%      & Numerical Contradiction&276\\
%      \bottomrule
%     \end{tabular}
%     \end{adjustbox}
%     \caption{Number of samples in NLI4CT dataset.}
%     \label{tab:dataset}
% \end{table}

To assess the performance of the models in the shared task, two metrics have been proposed: \textbf{Faithfulness} and \textbf{Consistency} \cite{jullien-etal-2024-semeval}, besides F1-score. Faithfulness measures the model's ability to adjust its predictions accurately after a semantic-altering intervention. Consistency measures the capacity of the system to predict the same label for both the original and contrast statements, in the case of semantic-preserving interventions.

