% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage{ACL2023}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{booktabs}
\usepackage{multirow} % for multirow command
\usepackage{graphicx}
\usepackage{natbib}


% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}


% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{SU-FMI at SemEval-2024 Task 5: From BERT Fine-Tuning to LLM Prompt Engineering - Approaches in Legal Argument Reasoning}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Kristiyan Krumov \\
  FMI, Sofia University \\
  \texttt{kristiyan.boyanov@gmail.com} \\\And
  Svetla Boytcheva \\
  Ontotext\\
   \texttt{svetla@uni-sofia.bg} \\
  \texttt{svetla.boytcheva@ontotext.com} \\\And
  Ivan Koytchev \\
  FMI, Sofia University \\
  \texttt{koychev@fmi.uni-sofia.bg} \\} 

\begin{document}
\maketitle
\begin{abstract}

This paper presents our approach and findings for SemEval-2024 Task 5, focusing on legal argument reasoning. We explored the effectiveness of fine-tuning pre-trained BERT models and the innovative application of large language models (LLMs) through prompt engineering in the context of legal texts. Our methodology involved a combination of techniques to address the challenges posed by legal language processing, including handling long texts and optimizing natural language understanding (NLU) capabilities for the legal domain. Our contributions were validated by achieving a third-place ranking on the SemEval 2024 Task 5 Leaderboard. The results underscore the potential of LLMs and prompt engineering in enhancing legal reasoning tasks, offering insights into the evolving landscape of NLU technologies within the legal field.

% In this paper we present investigation on the various transformer based workflows and different enhancements with large language models (LLMs) that aim to tackle the Task 5: The Legal Argument Reasoning Task in Civil Procedure of SemEval-2024 challenge. The porposed solutions show 
%This paper can be up to 5 pages excluding acknowledgments, references, and appendices. You can add an additional page for camera ready submission.

%You have to use the title as above, just replace "<Team Name>" and "<Descriptive Title>". Usual patterns are to use you team's TIRA code name as "<Team Name>" or to start "<Descriptive Title>" with "The <TIRA code name> approach [to/of/...]".

%At SemEval, papers are not anonymous when submitted for review.

%Your paper should focus on:
%\par\noindent{\bf Replicability}: present all details that will allow someone else to replicate your system. Provide links to code repositories if you made your code open source, and the docker image name if you used Docker submission. {\bf Note:} We will in our overview paper and at other opportunities point out which approaches are available open source and (even better) as Docker image to promote their widespread usage. If you re-submit your approach as Docker image in TIRA until the camera-ready deadline (and it produces the same results), please tell us so that we can include it in our overview paper.
%\par\noindent{\bf Analysis}: focus more on results and analysis and less on discussing rankings; report results on several runs of the system (even beyond the official submissions); present ablation experiments showing usefulness of different features and techniques; show comparisons with baselines.
%\par\noindent{\bf Duplication}: cite the task description paper \cite{kiesel:2023}; you can avoid repeating details of the task and data, however, briefly outlining the task and relevant aspects of the data is a good idea. (The official BibTeX citations for papers will not be released until the camera-ready submission period; the current bibtex entry is a placeholder and we will send you the correct one later.)

\end{abstract}

\section{Introduction}
Legal texts, including laws, interpretations, arguments, and agreements, are commonly conveyed through writing, resulting in great amount of legal documents. Analyzing these documents, a core aspect of legal work, becomes more intricate as these collections expand. Natural language understanding (NLU) technologies offer potential assistance to legal professionals in this regard. However, their effectiveness hinges on the ability of current state-of-the-art models to adapt to diverse tasks within the legal field.

The legal argument reasoning task \cite{Bongard.et.al.2022.NLLP} of SemEval-2024 represents a significant challenge in the domain of natural language processing (NLP) and an informal addition to the currently existing model evaluation benchmarks such as LexGLUE \cite{chalkidis-etal-2022-lexglue}.

Our approach involves fine-tuning pre-trained BERT models and exploring the innovative use of large language models (LLMs) through prompt engineering to address this task. 

% This paper discusses the importance of the task, our strategy encompassing transformer models and LLMs, and our key findings, including system performance and comparative analysis with other participants.

As a result of our work, we are ranked 3-rd in the SemEval 2024 Task 5\footnote{\url{https://trusthlt.github.io/semeval24/}} Leaderboard out of 20 participating teams. The implementations of the different approaches is available on Github\footnote{\url{https://github.com/frisibeli/semeval-2024-task5}} and the fine-tuned models could be accessed in Huggingface\footnote{\url{https://huggingface.co/frisibeli}}.

\section{Background}

Task 5 of SemEval 2024 is novel NLP problem focused on legal argument reasoning within the context of U.S. civil procedure. It contributes a dataset comprised of instances each containing a general introduction to a case, a specific legal question, a proposed solution argument, and a detailed analysis explaining the applicability of the argument. This dataset aims to benchmark the performance of legal language models, posing a significant challenge due to the complexity and nuanced understanding required for legal reasoning.
Instances are organized to support a binary classification task: determining the correctness of a given answer to a legal question, aimed at facilitating research on legal argument reasoning.

In the domain of text classification, conventional methodologies often employ "short encoders" such as BERT \cite{devlin2018bert}, RoBERTa \cite{liu2019roberta}, and XLNet \cite{yang2019xlnet}, which have demonstrated commendable efficacy in diverse contexts, ranging from news topic classification to sentiment analysis in movie reviews. Nevertheless, these encoders are constrained by their 512-token processing limit, rendering them less effective for analyzing extensive documents like court judgments. To circumvent this limitation, more advanced approaches, including the Hierarchical Attention Network (HAN) \cite{yang2016hierarchical}, a synergy of BERT and CNN, and the combination of XLNet with BiGRU \cite{chenxi2022research}, have been developed, enhancing the semantic understanding of longer texts. Despite these technological strides, the pursuit of an optimal algorithm that can adeptly navigate the complexities of extended documents persists.

The application of automated systems in the legal domain encounters distinct challenges, arising from the specialized language employed and the necessity for intricate multi-step reasoning over extensive texts. Furthermore, the potential of leveraging recent advancements in prompting techniques for legal domain-specific tasks remains largely unexplored. Typically, effective prompting in general NLP tasks has been noted with concise inputs, often limited to a single sentence or a small collection of sentences, accompanied by a restricted array of target labels. This underscores the ongoing quest to adapt and refine NLP techniques to meet the unique demands and intricacies of legal reasoning.



\section{System Overview}
\begin{figure*}[h]
    \centering
    \includegraphics[width=1\linewidth]{Blank diagram - Page 1 (1).png}
    \caption{Transformer-based classifier system architecture}
    \label{fig:transformer-classifier}
\end{figure*}
After analyzing the dataset, we identified that the final system should be capable of handling relatively lengthy contexts and to perform well on reasoning and fact-checking tasks. In this section we separately introduce the different approaches we have experimented with on solving the Legal Argument Reasoning task by dividing them into methods for handling long texts and such for optimizing the NLU capabilities for the legal domain.

\subsection{Handling Long Texts}
Observing the distributions (Fig. \ref{fig:data-length-tokens}) of the token lengths for the dataset entries we could say that a system capable of processing contexts of 2000 tokens would be sufficient to cover the majority of the cases. 
\begin{figure*}[h!]
    \centering
    \includegraphics[width=0.8\linewidth]{Screenshot 2024-02-17 at 9.34.46.png}
    \caption{Token count distribution of  the dataset per entry parts - Question, Explanation, Answer and Concatenated}
    \label{fig:data-length-tokens}
\end{figure*}

\subsubsection{Sliding Window (SW)}
We leveraged the sliding window techniques as described in \cite{Bongard.et.al.2022.NLLP}, as a baseline to overcome the maximum token limit problem. We experimented with \textbf{Sliding Window Simple} %(concatenated questions and introduction are split into chunks and an average of the predicted outputs is taken as a result)
and \textbf{Sliding Window Complex} 
%(similar to the previous, but the question is always present in the context).

\subsubsection{Transformer-based models for long text}\label{sec:longformers}
Transformer-based models encounter difficulty processing lengthy sequences due to their self-attention operation, which exhibits quadratic scaling with sequence length. In response to this constraint, we experimented with the Longformer \cite{beltagy2020longformer} model, featuring an attention mechanism that scales linearly with sequence length and increases the maximum input length to 4096 sub-word tokens, which may also improve the performance in understanding legal documents. Additionally, we experimented with Legal-RoBERTa and Legal-Longformer - pre-trained models on legal corpus introduced in \cite{chalkidis2023lexfiles}.

% \subsubsection{Hierarchical Implementations of BERT}

% Due to the less favorable outcomes obtained from the models outlined in Section \ref{sec:longformers}, we expanded our investigation by researching a hierarchical transformer variant such as as HIER-BERT \cite{chalkidis-etal-2022-lexglue}, \cite{chalkidis-etal-2019-neural} \cite{chalkidis2022exploration}. 
% The system works as initially, multiple BERT embeddings are created for the input text and subsequently, a self-attention mechanism analyzes these embeddings, generating a singular case embedding that is used for the classification task.

\subsubsection{Summarizing}
\begin{figure}[h]
    \centering
    \includegraphics[width=1\linewidth]{Screenshot 2024-02-17 at 13.24.56.png}
    \caption{Preprocessing an Explanation from the dataset using T5 for summarization}
    \label{fig:summrization}
\end{figure}
A different approach we tried for preprocessing lengthy texts was utilizing summarization models. By condensing extensive content into concise summaries, we not only mitigate the challenges posed by the length limitations of Transformer-based architectures but also streamline subsequent processing stages by reducing the presence of extraneous or tangential content, such as author's thoughts and remarks (Fig.  \ref{fig:summrization}).

As part of our solution, we examined several summarization models - BART \cite{DBLP:journals/corr/abs-1910-13461}, LexRank \cite{Erkan_2004} and ChatGPT4\footnote{\url{https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo}}.

\subsection{Optimising NLU Capabilities for the Legal Domain}
Research has demonstrated the efficacy of language model pre-training in enhancing numerous natural language understanding tasks like natural language inference \cite{devlin2019bert}. In addition to learning linguistic knowledge, these models are retaining relational knowledge \cite{petroni2019language} present in the training data which could be beneficial in solving downstream tasks in domains such as the legal one and more precisely - US Civil Procedure where the legal system is based on precedents. 
In this section we are going to reflect on the methods used by us to improve the performance of the system by enhancing its reasoning capabilities.

\subsubsection{Pre-trained Transformer Models on Legal Corpus}\label{sec:models}
As a starting point in addressing the problem, we decided to use Legal-BERT \cite{chalkidis2020legal}, being the most successful baseline experiment described in the work of the organizers of the task \cite{Bongard.et.al.2022.NLLP}. We fine-tuned it on the task and additionally - on a custom legal dataset \ref{sec:custom-dataset}.
Our contribution continued with the exploration of alternative legal transformer models: CaseHold-BERT \cite{zheng2021does}, variants of Legal-BERT (small, large), Legal-RoBERTa \cite{chalkidis2023lexfiles} and InLegalBERT \cite{paul2023pretrained}.

\subsubsection{\textbf{Fine-tuned BERT on Custom Dataset}}\label{sec:custom-dataset}
We additionally fine-tuned the best performing models from \ref{sec:models} on a custom-tailored dataset of an American civil procedure data (\ref{sec:custom-legal-dataset}), similar to the entries from the task. The goal with this approach was to strengthen the model's relational knowledge and contextual representations of the language used in the legal domain \cite{petroni2019language}. 

\subsection{\textbf{LLM + Legal Prompt Engineering}}

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{Blank diagram - Page 2.png}
    \caption{Generative model-based system architecture}
    \label{fig:generative-classifier}
\end{figure*}
So far we observed the task as a supervised classification problem, where the models are trained with labeled data to classify inputs into a binary output. Another approach is to use the relatively new method of prompt engineering in combination with some of the currently best-performing generative models (Fig. \ref{fig:generative-classifier}). With prompting, there's generally no need for additional training as the model receives a prompt, which could be a question, examples of input-output pairs (few-shot learning), or task descriptions. This approach allows the model to leverage its pre-trained knowledge to produce outputs for specific tasks in a zero-shot manner, meaning it can generate correct responses without having seen examples of the specific task during its training phase.
For this setup, we experimented with several types of LLMs: Mistral-7b-Instruct \cite{jiang2023mistral}, Llama2-70b \cite{touvron2023llama}, GPT-3.5-Turbo and GPT-4 \footnote{\url{https://platform.openai.com/docs/models}}; as for most of those models we performed prompt fine-tuning and Legal prompt engineering \cite{trautmann2022legal}.


\section{Experimental Setup}

\subsection{Data}
For the transformer-based classifier systems (Fig. \ref{fig:transformer-classifier}) we performed experiments on the SemEval 2024 Task \cite{Bongard.et.al.2022.NLLP} dataset. We stratified the train partition (750 entries) into train* (88\%) and train-dev (12\%), ensuring that the distribution of label values was maintained. The dev partition (84 entries) and the test partition were solely utilized for validation to prevent overfitting and bias in the model.

On other hand, for the generative-based classifier systems (Fig. \ref{fig:generative-classifier}) we used only the dev and test partitions leveraging the generalization capabilities of the large models and inferring in a zero-shot manner.


\subsection{Fine-Tuning \& Hyperparameters}
All experiments related to transformer-based classifier systems (Fig. \ref{fig:transformer-classifier}) were conducted using a single A100 40GB GPU, with the following hyperparameters: 5 training epochs and a learning rate of \(2e-5\). Additionally weight-decay and early-stopping (patience = 3) were applied.

For the generative-based systems different environments were used:
\begin{itemize}
    \item Local setup (Apple M2) + OpenAI access for GPT-4, GPT-3.5-turbo
    \item Local setup (Apple M2) including Ollama\footnote{\url{https://ollama.com/}} for running Mistral-7b and Llama2
\end{itemize}
We experimented with low temperature hyper-parameter values, ranging \(0-0.2\), in order to achieve more deterministic results.

\subsection{Custom Legal Dataset}\label{sec:custom-legal-dataset}
For MLM fine-tuning the transformer classifier, a new custom-tailored U.S. Civil Procedure dataset (Ref. \ref{sec:custom-dataset}) was used. It was collected first by automatically extracting the keywords from each unique explanation+question entry, then manually creating search queries and finally - using the open search API of the Caselaw Access Project\footnote{\url{https://case.law/}} downloading relevant cases.
The final corpora consists of 1985 different legal texts (cases), sourced by storing each 20 most relevant results for 100 queries.

\subsection{Legal Prompt Engineering (LPE)}
In \cite{trautmann2022legal}, the authors define "Legal prompt engineering (LPE)" as the process of creating, evaluating, and recommending prompts for legal NLP tasks. In the current work as an alternative approach to the transformer-based classifier systems we investigate the performance of LPE on the SemEval 2024 Legal task. 
We used more than 15 prompts (\ref{sec:prompts}), as for their creation, we followed some of the 26 principles described in \cite{bsharat2024principled}. Modification of a prompt version was done after evaluating how certain changes affect the performance.

The general frame of the prompt was in the form of a task or question, for which the model has to answer only with "TRUE" or "FALSE".  An interesting observation is that GPT-3/4 and Llama2 almost always follow that restriction and return one of the two desired outputs with very few times returning something slightly different (e.g. different casing or appending punctuation - "false.", "True"). Contrarily, Mistral-7b-instruct always returns the answer with an additional explanation, which led to a more complex post-processing step for that model.

We used LangChain\footnote{LLM framework - \url{https://python.langchain.com/}} for prompt template processing, model-agnostic interface unification and easy response post-processing.  

\section{Results}

\begin{table*}[h] % Use htbp as placement specifiers, adjust as needed
    \centering
    \begin{tabular}{|l|l|c|c|}
        \hline
        \textbf{Model Name} & \textbf{System Type} & \textbf{Dev Macro-F1} & \textbf{Test Macro-F1} \\
        \hline
         Majority& \multirow{2}{*}{Baseline}& 0.44& 0.42\\
        Random& & 0.46& 0.46\\
        \hline
        CaseHold/LegalBERT + SW& \multirow{6}{*}{Transformer} & 0.55& -\\
        LegalBERT + SW& & 0.59& -\\
        LegalBERT-small + SW& & 0.53& -\\
        InLegalBERT + SW& & 0.44& -\\
        lexlms/legal-longformer-base& & 0.50& -\\
        SU-FMI-LegalBERT + SW& & 0.60& -\\
        lexlms/legal-roberta-large& & 0.62& 0.49\\
        \hline
        legal-roberta + BART& \multirow{4}{*}{Classifier + Summary}& -& 0.50\\
        SU-FMI-LegalBERT + BART& & -& 0.52\\
        CaseHold/LegalBERT + BART& & -& 0.54\\
        CaseHold/LegalBERT + GPT-4& & -& 0.55\\
        CaseHold/LegalBERT + GPT-4 V2& & -& 0.61\\
        \hline
        Mistral-7b + LPE& \multirow{4}{*}{LLM}& -& 0.58\\
        Llama2-70b + LPE& & 0.59& 0.58\\
        GPT-3.5-turbo + LPE& & 0.58& 0.60\\
        GPT-4 + LPE& & 0.74& 0.7728\\
        \hline
    \end{tabular}
    \caption{Model Performance on Development and Test Sets}
    \label{tab:model_performance}
\end{table*}

% \begin{table*}[h] % Use htbp as placement specifiers, adjust as needed
%     \centering
%     \begin{tabular}{|l|l|c|c|c|c|}
%         \hline
%         \multirow{2}{*}{\textbf{Model Name}} & \multirow{2}{*}{\textbf{System Type}} & \multicolumn{2}{c|}{\textbf{Dev}} & \multicolumn{2}{c|}{\textbf{Test}} \\
%         \cline{3-6}
%         & & \textbf{Macro-F1} & \textbf{Accuracy} & \textbf{Macro-F1} & \textbf{Accuracy} \\
%         \hline
%         casehold/legalbert + SW& Transformer& 0.55& -& -& -\\
%         legal-bert + SW& Transformer& 0.59& -& -& -\\
%  legal-bert-small + SW& Transformer& 0.53& -& -&-\\
%  InLegalBERT + SW& Transformer& 0.44& -& -&-\\
%         lexlms/legal-longformer-base& Transformer& 0.50& -& -& -\\
%  custom-legal-bert + SW& Transformer& 0.60& -& -&-\\
%  lexlms/legal-roberta-large& Transformer& 0.62& -& 0.49&-\\
%         \hline
%         legal-roberta + bart (summary)& Summary & -& -& 0.50& -\\
%  custom-legal-bert + bart& Summary & -& -& 0.52&-\\
%  casehold/legalbert + bart& Summary & -& -& 0.54&-\\
%  casehold/legalbert + gpt4 (summary)& Summary & -& -& 0.55&-\\
%  casehold/legalbert + gpt4 V2& Summary & -& -& 0.61&-\\
%         \hline
%         Mistral-7b + LPE& Generative Model & -& -& 0.58& 0.63\\
%         llama2-70b + LPE& Generative Model & 0.59& -& 0.58& -\\
%  GPT-3.5-turbo + LPE& Generative Model& 0.58& -& 0.60&-\\
%         GPT-4 + LPE&  Generative Model & 0.74& -& 0.7728& 0.8367\\
%         \hline
%     \end{tabular}
%     \caption{Model Performance on Development and Test Sets}
%     \label{tab:model_performance}
% \end{table*}


Table \ref{tab:model_performance} shows the results of the different experiments. The evaluation of different models for the SemEval-2024 Task 5 on Legal Argument Reasoning presents interesting observation on how different models and system types, described in the current work perform on the dev and test dataset partitions. Our baseline approaches, Majority and Random, set the initial benchmarks with Macro-F1 scores significantly lower than those achieved by advanced models, underscoring the complexity of the task. The application of transformer models, including those equipped with a Sliding Window technique and summarization capabilities, such as BART, showed improvement over the baselines, indicating the value of contextual understanding and content summarization in legal reasoning tasks.

Notably, the integration of Large Language Models (LLMs) with Legal Prompt Engineering (LPE) techniques, particularly with GPT-3.5-turbo and GPT-4, led to a significant leap in performance metrics. These models outperformed traditional transformer models, highlighting the effectiveness of LPE in enhancing the model's ability to interpret and reason over legal texts.

The comparative analysis of model performances on both development and test datasets revealed consistent patterns. Models utilizing LLMs with LPE not only achieved the highest Macro-F1 scores but also demonstrated robustness across different data sets, underscoring their potential for real-world applications in legal reasoning and argumentation.


\section{Conclusion}
Our participation in SemEval-2024's legal argument reasoning task has yielded valuable insights into the capabilities of transformer-based models and LLMs in processing and reasoning over legal texts. While our methods have shown promise, particularly in leveraging LLMs and prompt engineering, the complexity of legal reasoning poses ongoing challenges. 

Further investigation can be done in a solution based on a hierarchical transformer variant such as HIER-BERT \cite{chalkidis-etal-2022-lexglue}, \cite{chalkidis-etal-2019-neural} \cite{chalkidis2022exploration}. Our initial experiments with that model architecture did not lead to very high results (0.47 f1-macro on the dev partition) and because of the setup complexity, we decided to leave it for future research opportunities.

\section{Acknowledgments}
This work was partly supported by the European Union-NextGenerationEU, through the National Recovery and Resilience Plan of the Republic of Bulgaria, project No BG-RRP-2.004-0008.


\bibliography{custom} 
\bibliographystyle {acl_natbib}


\appendix
\section{Appendix}
\subsection{Prompts} \label{sec:prompts}
\subsubsection{Best Prompt (GPT-4, GPT-3.5, Llama2-70b}
\textbf{System Prompt:}

\textit{system\_prompt = """}

\textit{You are a legal assistant with a specialization in U.S. Civil Procedure. Your role involves thorough analysis and resolution of cases pertaining to this field. You will encounter three key components in each case:}

\textit{1. EXPLANATION: This provides additional context and background information about a specific lawsuit.}

\textit{2. QUESTION: Here, you will be presented with actual facts and details surrounding the lawsuit.}

\textit{3. HYPOTHESIS: Based on the provided information, a hypothesis will be presented. Your task is to rigorously evaluate this hypothesis in the context of U.S. Civil Procedure and determine its validity. Respond ONLY with 'TRUE' if you conclude that the hypothesis is correct, or ONLY with 'FALSE' if you find it to be incorrect. }

\textit{Do no}t provide any reasoning behind your decision.

"""

\textbf{User Input:}

\textit{input\_template = """}

EXPLANATION: \{\}

QUESTION: \{\}

HYPOTHESIS:  \{\}

\textit{"""}

\subsubsection{Chain of thoughts}

system\_prompt = """

You are a legal assistant with a specialization in U.S. Civil Procedure. Your role involves thorough analysis and resolution of cases pertaining to this field. You will encounter three key components in each case:

1. EXPLANATION: This provides additional context and background information about a specific lawsuit.

2. QUESTION: Here, you will be presented with actual facts and details surrounding the lawsuit.

3. HYPOTHESIS: Based on the provided information, a hypothesis will be presented. Your task is to rigorously evaluate this hypothesis in the context of U.S. Civil Procedure and determine its validity. 

On User input with EXPLANATION, QUESTION and HYPOTHESIS analyse the legal problem step by step. Explain your thoughts.

"""

final\_input = """

Respond ONLY with 'TRUE' if you conclude that the hypothesis is correct, or ONLY with 'FALSE' if you find it to be incorrect. 

Do not provide any reasoning and ONLY answer with 'TRUE' or 'FALSE'

"""

\subsubsection{Mistral-7b-instruct Best Prompt}
You are a helpful civil law assistant. Your answer only with "TRUE" or "FALSE". You answer with "TRUE" if the STATEMENT is correct based on the provided CONTEXT or "FALSE" otherwise. If you don't know the answer - answer with FALSE. \\
=====================\\
The CONTEXT is \{explanation\} | \{question\} \\
=====================\\
The STATEMENT is: \{answer\}



\end{document}
