% We would like to emphasize the importance of analysis of your system’s behavior beyond the overall scores. There will be a best paper award to recognize system papers with strong analysis.

\begin{table}[hbt!]
  \begin{adjustbox}{width=0.5\textwidth}
  \begin{tabular}{lcccc}
    \toprule
      \textbf{Model} & \textbf{Setting} & \textbf{F1} & \textbf{Faithfulness} & \textbf{Consistency} \\
    \midrule
    \texttt PubMedBERT & fine-tuned & 0.63 & 0.53 & 0.62 \\
    \texttt XLM-RoBERTa & fine-tuned & 0.63 & 0.55 & 0.63 \\
    \texttt Deberta V3 Large & fine-tuned & 0.63 & 0.54 & 0.62 \\
    \texttt BioBERT & fine-tuned & 0.65 & 0.52 & 0.63 \\    
    \midrule
    \texttt SciFive Pubmed PMC &zero-shot & 0.56 & 0.61 & 0.56 \\
    \texttt Llama-2 7B & zero-shot & 0.65 & 0.19 & 0.44 \\
    \texttt Mistral 7B & zero-shot & 0.65 & 0.18 & 0.44 \\
    \texttt Mistral 7B Instruct-0.2 & zero-shot & \textbf{0.72} &  0.68 & \underline{0.66} \\
    \texttt SOLAR 10B * & {zero-shot} & 0.63 & \textbf{0.90} & \textbf{0.72} \\
    \midrule
    \texttt SOLAR 10B & {few-shot}  & \underline{0.71} & \underline{0.83} & \textbf{0.72} \\
    \bottomrule
  \end{tabular}
  \end{adjustbox}
    \caption{Results of our submissions for the SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. Best results are presented in \textbf{bold}, and second-best results are presented in \underline{underline}. Results with asterisk (*) were not submitted.}
      \label{tab:official-results}
\end{table}

The official results of our team can be found in Table \ref{tab:official-results}. Our results indicate that using pre-trained models on clinical text does not significantly improve the performance on this particular task, despite research confirming that domain-specific language model pre-training can be beneficial for other biomedical tasks \cite{Tinn2022}. In line with last year's findings \citep{jullien-etal-2023-semeval}, we observe that instruction-tuned models pre-trained on generic datasets perform better than discriminative models pre-trained on biomedical datasets. With respect to LLaMa-2 and Mistral models used in zero-shot settings, they achieve high F1 scores of 0.65 and 0.72. However, these models are not robust enough and achieve low performance on Faithfulness and Consistency metrics, which are the metrics the organizers focused on. We further expand on the results of our best-performing model, SOLAR. 


% Control set: comparativ cu anul trecut, F1 ar fi în top 5, însă cu un model mult mai mic (fiind cuantizat)
% și folosind 2-shot, fără fine-tuning
% SOLAR este < 1/2 din Flan-T5; nu merge comparat cu Flan-T5 de anul trecut pt că avem rezultate pe test set
% doar cu few-shot, iar ei fac doar 0-shot



\textbf{Control set.} We obtain an F1-score of 0.71, with the highest score for comparisons of adverse events (0.79 F1)
and the lowest score for comparisons of interventions (0.62 F1). Our team reaches the {16th} place out of 32 participants in the official leaderboard. Compared to last year, we would be ranked on {5th} place while using little to no training data and modest computational resources. Similar to last year, we report a higher Recall (0.73) than Precision (0.70).

\textbf{Contrast set.} Our system achieves a Faithfulness score of 0.83 (10th place out of 32 teams) and a Consistency
score of 0.72 (14th place out of 32 teams). This shows that the model is more reliable when dealing with semantic-altering transformations compared to semantic-preserving changes, with only one CTR section having a faithfulness score below 0.79 (eligibility comparisons - 0.71). We observe a similar behavior in 22 other submissions where faithfulness
is higher than consistency.

\section{Error Analysis and Discussion}
\label{sec:discussion}
% \subsection{??}

In this section, we analyze the types of errors made by the SOLAR model according to the provided metrics based on each intervention target. % We omit from this analysis systematic errors occurring due to implementation defects.
% 345 examples appear to be affected by splitting the hypothesis wrong; with a partial fix, we see a ~0.5 improvement in F1 and consistency; also not submitted

% Exemplul b23563cf-e6c4-4279-92b7-39d5e4de96d2 pare greșit:
% statement: None of the cohort of the primary trial or the secondary trial receive enteric-coated tablets
% secondary CT: MLN8237 (alisertib) 10 milligram (mg), enteric-coated tablets...
% label: Entailment
% our prediction: Contradiction

% alt exemplu care pare greșit?? sau cel puțin ambiguu: 94ebad4c-4b9c-4fb7-b7cc-a088b46b56f7
% modelul prezice label-ul din gold dacă punem MCS în loc de mcs - am zis deja de abrevieri

% Suggestions for examples to focus on:
% - Particular examples where the system makes errors, especially if they seem systematic
  % - systematic error due to implementation bug: some summaries are extremely short or do not contain the relevant information: this affects especially the eligibility section scores
  % confusions between inclusion and exclusion criteria for eligibility sections: ``Sleep disruption or high insomnia severity index score. Exclusions include \[...\]''. Insomnia is mentioned before exclusion criteria, leading to 
  
% - Differences in types of errors between models
% - Unsuccessful approaches and attempts: 
     % - fine-tuning MLMs (Claudiu): ...
     % - BioGPT, MedAlpaca (Marius); phi-2 nu a mers printre altele din cauză că nu era instruction fine-tuned
     % - sensitivity to minor changes in prompt (spaces) (Marius)
     % - ....

% Behavior on inputs with the characteristics emphasized in task description:


% Intervention targets:

% - Numerical - LLMs still struggle to consistently apply numerical and quantitative reasoning. As NLI4CT requires this type of inference, we will specifically target the models' numerical and quantitative reasoning abilities.

% - Vocabulary and syntax - Acronyms and aliases are significantly more prevalent in clinical texts than general domain texts, and disrupt the performance of clinical NLI models. Additionally, models may experience shortcut learning, relying on syntactic patterns for inference. We target these concepts and patterns with an intervention.

% - Semantics - LLMs struggle with complex reasoning tasks when applied to longer premise-hypothesis pairs. We also intervene on the statements to exploit this.


\textbf{Definition interventions.} These interventions simply append a sentence to the statement. The model is capable of extracting the relevant sentence containing references to clinical trials if asked explicitly through a separate pre-processing step, but this incurs an additional runtime cost. Ultimately, we tackle this issue with a simple heuristic (see the inference details in \ref{sec:llms}).
% the heuristic was not implemented properly, which led to bogus splits: e.g. 0.3cm -> 3cm

% Definition interventions: aici modelul s-a descurcat bine
% TODO: de văzut impactul sumarizării și 0-shot vs few-shot; de căutat 1-2 exemple pe care nu s-a descurcat
% Definitions_Consistency:  0.7046666666666667
% Definitions_F1:  0.6955326460481099
% Definitions_Recall:  0.7177304964539007
% Definitions_Precision:  0.6746666666666666

% de văzut pe fiecare secțiune în parte
% cel mai prost a dat pe comparison_eligibility, comparison_intervention și single_adverse_events


\textbf{Numerical interventions.} Even though the SOLAR model has been tuned for mathematical reasoning, we identified several shortcomings. The model performs poorly with measurement units that express the same quantity in different ways. It appears to understand the meaning of symbols (e.g. ``positive'' instead of ``+''), but if domain-specific acronyms are lowercase instead of uppercase (e.g. ``hr'', ``her2'', ``mcs'', ``pdr''), the prediction changes.

Another interesting example is related to how entailment is affected when numbers have similar semantic meanings in other contexts. In one of the intervention trials, it is specified that a treatment cycle takes 21 days. The statement asks whether the trial has a 30-day cycle, with the model classifying it as entailment. Slightly tweaking the number in the statement reveals that this is not regarded as entailment if the values do not match semantically: only the pairs 21-28, 21-30 and 21-31 are considered entailment; the other pairs from 21-26 to 21-34 are classified as contradiction. Nonetheless, if we change the number in the trial (21) instead of the statement, it always predicts an entailment, even when the value is nonsensical. This appears to be a drawback of long contexts as the issue only manifests with 2-shot prompting.

For numerical paraphrasing, we have 0.67 Consistency, 0.54 F1-score, 0.63 Recall and 0.47 Precision. Conversely, for numerical contradictions, the model obtains 0.80 Faithfulness, 0.85 Consistency, 0.91 F1, 1.0 Recall and 0.83 Precision\footnote{We adapted the evaluation script to use \texttt{pos\textunderscore label} as 0 for contradictions and numerical contradictions.}.

% Numerical_Para_Consistency:  0.6741071428571429
% Numerical_Cont_Faithfulness:  0.8070175438596491
% Numerical_Cont_Consistency:  0.8580246913580247
% Numerical_Para_F1:  0.5408805031446541
% Numerical_Para_Recall:  0.6323529411764706
% Numerical_Para_Precision:  0.4725274725274725
% Numerical_Cont_F1:  0.911242603550296            % cu pos_label=0
% Numerical_Cont_Recall:  1.0                      % cu pos_label=0
% Numerical_Cont_Precision:  0.8369565217391305    % cu pos_label=0

% numerical contradictions - erorile nu par să fie (doar) din cauza alterațiilor numerice
% cf confusion matrix nimerește majoritatea exemplelor, dar script-ul nu e adaptat corect:
%   contradictions au doar gold labels negative și f1_score e by default average='binary' cu pos_label=1,
%   dar datele au doar pos_label=0

\textbf{Paraphrasing and contradiction interventions.} We notice a similar behavior to the numerical interventions: very high scores for contradictions, but significantly lower results for rephrases. Thus, paraphrases have 0.70 Consistency, 0.69 F1, 0.72 Recall, and 0.66 Precision, while contradictions have 0.83 Faithfulness, 0.78 Consistency, 0.89 F1, 1.0 Recall, and 0.80 Precision.

Our results suggest the model is overly sensitive to any wording change between the hypothesis and premise, mistakenly interpreting them as conflicting. This also explains the perfect recall scores.

% Para_Consistency:  0.7033333333333334
% Cont_Faithfulness:  0.8346666666666667
% Cont_Consistency:  0.7813333333333333
% Para_F1:  0.6903270702853167
% Para_Recall:  0.7219796215429404
% Para_Precision:  0.6613333333333333
% Cont_F1:  0.8938053097345133    % cu pos_label=0
% Cont_Recall:  1.0               % cu pos_label=0
% Cont_Precision:  0.808          % cu pos_label=0

% \subsection{Discussion}
In the remainder of this section, we present a few high-level remarks related to our design decisions.

% - 2-shot în general ajută și e o îmbunătățire față de zero-shot, indiferent de configurație; cu 6-shot se degradează performanța, cel puțin cu modelele și prompts încercate de noi; task-ul este dificil: deși nu am atins limita maximă de 4096 tokens a modelului, modelul nu pare să se descurce cu mai multe exemple în același prompt
% dă mai prost cu few-shot decât cu zero-shot pe exemple lungi - eligibility comparisons

% \textbf{Few-shot prompting consistently improves results compared to zero-shot prompting.}
% We used two examples from the training set, an entailment and a contradiction. This approach improved the results on all available metrics, regardless of model, model size or summarization configuration. Adding more examples to few-shot prompting might further increase the results at the expense of slower inference speed. However, with 6-shot prompting, the performance degrades even if we do not reach the context limit of 4096 tokens. We assume this is caused by the complexity of the task and the model ``forgetting'' what task it must solve.
\textbf{Few-shot prompting might improve results compared to zero-shot prompting.}
We used two examples from the training set, an entailment and a contradiction. This approach improved the results on average on the development set in terms of F1, regardless of model, model size or summarization configuration. Adding more examples to few-shot prompting might further increase the results at the expense of slower inference speed. However, with 6-shot prompting, the performance degrades even if we do not reach the context limit of 4096 tokens. We assume this is caused by the complexity of the task and the model ``forgetting'' what task it must solve. We do not further explore 6-shot prompting because of poor preliminary performance and increased runtime on a subset of the development data.

On the test set, it appears that few-shot prompting degrades the results in terms of Faithfulness and Contradiction, although it improves paraphrasing scores. We argue that a higher Faithfulness score in itself does not imply better results because the model could simply predict more Contradictions when the input is altered.

% scoruri 0-shot
% Control_F1:  0.6370023419203747
% Control_Recall:  0.768361581920904
% Control_Precision:  0.544
% Contrast_F1:  0.573053679345894
% Contrast_Recall:  0.6595744680851063
% Contrast_Precision:  0.5065996228786926
% Faithfulness:  0.9085648148148148
% Consistency:  0.7287234042553191
% Para_Consistency:  0.684
% Cont_Faithfulness:  0.904
% Cont_Consistency:  0.8746666666666667
% Numerical_Para_Consistency:  0.6651785714285714
% Numerical_Cont_Faithfulness:  0.9385964912280702
% Numerical_Cont_Consistency:  0.9629629629629629
% Definitions_Consistency:  0.6846666666666666
% Para_F1:  0.6189710610932476
% Para_Recall:  0.7793522267206477
% Para_Precision:  0.5133333333333333
% Cont_F1:  0.941425546930134
% Cont_Recall:  1.0
% Cont_Precision:  0.8893333333333333
% Numerical_Para_F1:  0.41860465116279066
% Numerical_Para_Recall:  0.7105263157894737
% Numerical_Para_Precision:  0.2967032967032967
% Numerical_Cont_F1:  0.9758812615955473
% Numerical_Cont_Recall:  1.0
% Numerical_Cont_Precision:  0.9528985507246377
% Definitions_F1:  0.6249008723235527
% Definitions_Recall:  0.7710371819960861
% Definitions_Precision:  0.5253333333333333


% scoruri 2-shot
% Control_F1:  0.7116564417177914
% Control_Recall:  0.7280334728033473
% Control_Precision:  0.696
% Contrast_F1:  0.6176122931442081
% Contrast_Recall:  0.5828220858895705
% Contrast_Precision:  0.6568196103079824
% Faithfulness:  0.8310185185185185
% Consistency:  0.7224371373307543
% Para_Consistency:  0.7033333333333334
% Cont_Faithfulness:  0.8346666666666667
% Cont_Consistency:  0.7813333333333333
% Numerical_Para_Consistency:  0.6741071428571429
% Numerical_Cont_Faithfulness:  0.8070175438596491
% Numerical_Cont_Consistency:  0.8580246913580247
% Definitions_Consistency:  0.7046666666666667
% Para_F1:  0.6903270702853167
% Para_Recall:  0.7219796215429404
% Para_Precision:  0.6613333333333333
% Cont_F1:  0.8938053097345133
% Cont_Recall:  1.0
% Cont_Precision:  0.808
% Numerical_Para_F1:  0.5408805031446541
% Numerical_Para_Recall:  0.6323529411764706
% Numerical_Para_Precision:  0.4725274725274725
% Numerical_Cont_F1:  0.911242603550296
% Numerical_Cont_Recall:  1.0
% Numerical_Cont_Precision:  0.8369565217391305
% Definitions_F1:  0.6955326460481099
% Definitions_Recall:  0.7177304964539007
% Definitions_Precision:  0.6746666666666666








\textbf{Instruction order is significant.} We observed the best results when the instructions were placed at the
start of the prompt, followed by the CTR (or CTR summary) and then the hypothesis. This strategy constantly provides
improved predictions across most of the evaluated prompts. There is a drop of 4 F1 percentage
points on the development set when the hypothesis is placed before the CTR summary in the prompt template. Separately, the predictions are also affected if the instruction is placed at the end of the prompt, after the hypothesis. Repeating the instruction before and
after the CTR-hypothesis pair is as effective as simply placing the instruction before the CTR text.

% {\color{red} Comentat din lipsă spațiu - eventual pus în anexe}
% \textbf{Some predictions are sensitive to minor alterations.} While this SemEval task proposes higher-level semantic
% interventions, we find that punctuation, spaces, and letter casing can also influence the model prediction.
% In a similar fashion, the grammar used to restrict the model output affects a small number of examples. For
% instance, we might limit the output to ``\textbackslash nYes \textbar \ \textbackslash nNo'' instead of
% ``Yes \textbar \ No'' because the model typically starts the text generation with a space or newline character.


% - pe setul de dev nu e o diferență de scor dacă rezumatul e pe un nr de tokens mult mai mic; se pot obține scoruri mari chiar dacă datele sunt spurious (analizând, am observat că rezumatele nu conțin destule informații, mai ales în cazul exemplelor comparate), ceea ce înseamnă că metrica avută la dispoziție (F1) nu reflectă realitatea suficient de bine

% lăsăm acest paragraf având în vedere metricile folosite?
% ideea e că nu am evaluat prea mult pe test sets din lipsă de timp

% {\color{red} Comentat din lipsă spațiu - eventual pus în anexe}
% \textbf{Traditional metrics such as F1-score can be misleading.}
% % \textbf{High F1-scores can be misleading.}
% As the inference on the test sets is time consuming, many of our summarization experiments were conducted on
% the development set, since the F1-score of our initial submissions seemed to correlate with the faithfulness
% and consistency metrics. When the CTR summary length is reduced to 128 tokens, the model performs better in
% terms of F1-score, reaching 0.76 F1. On a closer inspection, we observe that many summaries do not actually
% provide information related to the hypothesis. One explanation for this fact is that the examples in the
% development set are not necessarily representative, varied or difficult enough for this task,
% allowing the model to learn spurious correlations. Another possible explanation is that the F1 metric is not
% enough by itself to assess the quality of these predictions.


% - nu pare să fie o diferență semnificativă între variații mici ale aceluiași model cu cuantizări diferite, deci am putea rula experimentele mai costisitoare pe modele mai mici fără o degradare mare a performanței; totuși, unele prompt-uri par sensibile la schimbarea modelului (de dat exemple)

\textbf{Larger models obtain better results, but smaller models are still useful for prototyping.} We use the
3-bit quantization version of the same SOLAR model (Q3\_K\_M) to experiment with more prompts, taking advantage
of faster inference times. This approach has been very useful in designing the summarization prompts because the
summarization step is the most expensive one in terms of computational resources. We also experimented with hybrid
systems, where the summaries are generated with a smaller model and the inference task is done by a larger model.
The performance of the hybrid strategy is comparable to running the entire pipeline with the larger model.


% - nu pare să fie atât de important/semnificativ ce prompt de sumarizare folosim, cât timp verificăm că în rezumatul rezultat sunt păstrate informațiile necesare pentru rezolvarea task-ului; modelul pare să se descurce și cu rezumate cu abrevieri

\textbf{There are no hard constraints for summaries prompts, as long as they do not depend on the hypothesis.}
There appears to be no substantial difference in the final results when changing the prompt used to generate CTR
summaries. We apply some of the following restrictions in each summarization prompt: use abbreviations, avoid verbs,
use short sentences, be brief, maximum N words. Rephrasing the prompt does not seem to have a relevant impact for
this task. As previously mentioned, it is paramount that the summaries preserve the meaning of the original text. The absence of relevant information in summaries is a major source of errors in our system.
Unfortunately, for most sections the model was unable to create contextualized summaries conditioned by
the hypothesis without mentioning the hypothesis in the summary.

For the Results section with a single CTR, conditioning the summary on the hypothesis looked promising.
However, the next example from the test set confirms our concerns about this strategy, where a single summary for multiple hypotheses is not appropriate due to conditioning on the first hypothesis in the dataset. Our generated summary is:  ``{There is no
information in the given CTR report that relates to the statement about all patients treated with GTx-024 1mg
gaining lean body mass over a 10 year period}''. The model is distracted by the ``10 year period'' from one of the hypotheses, altering the original meaning completely, even though the word ``year'' does not appear anywhere in the initial CTR section. See \cref{sec:mismatch-example} for the full CTR text and associated hypotheses.

