In this section, we present the different approaches used to predict the entailment relations.

\subsection{Pre-trained Masked Language Models}

Our first approach for the task of entailment relations prediction is using pre-trained models. Previous research has shown that domain-specific pre-training is beneficial for biomedical tasks \cite{Tinn2022,romanov2018lessons}.

We use pre-trained models, such as PubMedBERT \cite{gu2021domain}, XLM-RoBERTa \cite{conneau2020unsupervised}, Deberta V3 Large \cite{he2021deberta}, and fine-tune them on the training data. The fine-tuning process is done in 2 steps, as suggested in \cite{sun2020finetune}. Firstly, we stack a fully connected layer on top of the pre-trained model and train it for 4 epochs while the weights of the pre-trained model are frozen with a learning rate of $10^{-3}$. For the second step, we train the fully connected layer along with the last layer of the pre-trained model at a lower learning rate of $2*10^{-4}$ for one more epoch.

Inspired by the approach taken by \citet{pahwa2023bphigh}, who demonstrated that the performance of fine-tuned cross-encoders is comparable to that of GPT-3.5 in the few-shot setting, we experiment with the sentence-transformer model BioBERT\footnote{\href{https://huggingface.co/pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb}{pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb}} trained on 6 benchmark NLI datasets \cite{deka2022evidence} for sentence similarity tasks. We train a cross-encoder model using the sentence embeddings from BioBERT on the NLI4CT train data for sentence-pair classification for 20 epochs.

We also utilized SciFive \cite{phan2021scifive}, a T5 model designed for biomedical literature-related tasks. This model was pre-trained on a large corpus of PubMed abstracts and PubMed Central full-text articles from biomedical and life sciences domains. It achieved state-of-the-art results on the MedNLI benchmark dataset \cite{romanov2018lessons}. We used the SciFive model trained on MedNLI\footnote{\href{https://huggingface.co/razent/SciFive-large-Pubmed\_PMC-MedNLI}{razent/SciFive-large-Pubmed\_PMC-MedNLI}} in zero-shot setting to predict the entailment relations for the NLI4CT data.

\subsection{Large Language Models} \label{sec:llms}

LLMs have achieved promising results in biomedical tasks, such as named entity recognition, relation extraction, text classification, and question answering \cite{jahan2023comprehensive}. We conducted experiments using LLaMa-2 7B\footnote{\href{https://huggingface.co/meta-llama/Llama-2-7b}{meta-llama/Llama-2-7b}} \cite{touvron2023llama}, Mistral 7B\footnote{\href{https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2}{mistralai/Mistral-7B-Instruct-v0.2}} \cite{jiang2023mistral} and SOLAR \cite{kim2023solar}, which have shown promising results in various language tasks. LLaMa-2 is a competitive model that has performed well across multiple benchmarks such as commonsense reasoning, word knowledge, and reading comprehension. Mistral, on the other hand, has surpassed LLaMa-2 in all the tested benchmarks. We choose SOLAR-Instruct since it is a state-of-the-art model that is instruction-tuned specifically to have improved mathematical capabilities, with rephrased examples using a similar process to MetaMath \cite{yu2023metamath}. The team behind SOLAR developed a unique scaling technique, called depth up-scaling (DUS), which combines architectural changes with continued pretraining and obtained better results than larger models like Mixtral \cite{kim2023solar}. 

LLaMa-2 and Mistral were evaluated only in zero-shot settings on the NLI4CT test data. The same prompt was used in the experiments, regardless of the section the statement was referring to and it is presented in \Cref{sec:prompts}. While the experiments from LLaMa-2 and Mistral were using the entire sections of CTRs and statements, we had a different approach for SOLAR, which involved CTRs summarization. We expand on the methodology below.

\textbf{Section content summarization approach.} This approach consists of two stages. First, we summarize the text of each section to reduce the number of tokens.
Next, we perform the classification task on the shortened text using two-shot prompting with examples from the training set.
Both stages of this pipeline use the SOLAR 10.7B Instruct v1.0 model \citep{kim2023solar} in GGUF format\footnote{\href{https://huggingface.co/TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF}{TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF}} with 5-bit quantization (Q5\_K\_M) using \href{https://github.com/ggerganov/llama.cpp/tree/c9b316c78fba31e65879a2ec91cbafd341b88cce}{llama.cpp}.


% found using huggingface llm leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard


\textbf{CTR summarization.} We summarize the sections of each CTR in the evaluated datasets (train, development, test) to a maximum of 350 tokens.
The main reason for performing summarization is to provide the model with a shorter context and a task that is easier to solve. To validate this statement, we perform preliminary runs on the development set on Single CTRs for the Results section. We obtain an F1-score of 0.55 on single results CTR without a summary, while we are able to reach 0.63-0.72 F1-score with the summarization approach.

Another motivating factor is the time required to run the inference in order to perform multiple experiments. Full CTR inference for one example can take up to 20-30 seconds. Conversely, shortened CTRs are evaluated in roughly 5 seconds. As generating summaries is a one-time cost, the time difference is compensated for after a few iterations.
Evaluating on CTR summaries instead of the full CTR allowed us to dedicate more time in designing and refining prompts used for the entailment task. For these reasons, we do not conduct additional experiments on full CTRs.

The obvious drawback of summarization is the risk of discarding essential information. 
In the initial experiments for this approach, we tried to mitigate this by conditioning the summary to be related to the
hypothesis statement. Unfortunately, this strategy caused the model to include the statement in the summary and at times
even output contradictory phrases. Moreover, we did not try to continue with contextualized summaries because it would
require generating a new summary for each statement instead of a summary for each CTR section. Given that the evaluation
relies on using the same section with multiple statements, we need to generate only one summary per section if the summary
does not depend on the hypothesis.

\textbf{Inference.} We solve the entailment task through zero-shot and two-shot prompting.
As some statements might contain irrelevant sentences, we only keep the first sentence that contains the word ``trial''. We are aware that it is not an ideal approach and we lose important information in some examples where 2 or more sentences are crucial for the task. We found that the model works better if we keep only the ``trial'' sentence rather than not performing this step at all.
We also tried a more robust approach by asking the model to extract the relevant sentences from the hypothesis. While this tackles the issues encountered by our simple heuristic mentioned before, we limit our system setup to sentence splitting since it is significantly faster.

We limit the output grammar of the model to only ``Yes'' and ``No'' to ease processing. For each of the four sections and example types (Single or Comparison), we apply different
prompts for summarization and evaluation.
%summarization prompts and evaluation prompts.
The final prompt templates are listed in \Cref{sec:prompts}. % they take up a lot of space
The advantage of this approach is that we can analyze and tune the prompts independently for each section,
without running the inference step for the whole dataset.

% this is pretty much everything regarding system description
% the rest would only be failed/discarded attempts

% this would go in discussions or scrapped due to space??
%At first, we used a single prompt for evaluating all sections, reaching an F1-score of $\approx$ 74\%
% on the development set. 