% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\usepackage{graphicx}

\usepackage{adjustbox}

\title{Mistral at SemEval-2024 Task 5: Mistral 7B for Argument Reasoning in Civil Procedure}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Marco Siino \\
Department of Electrical, Electronic \\and
Computer Engineering \\
  University of Catania \\
  Italy \\
  \texttt{marco.siino@unipa.it} }

\begin{document}
\maketitle
\begin{abstract}


At the SemEval-2024 Task 5, the organizers introduce a novel natural language processing challenge and corpus within the realm of the United States civil procedure. Every datum within the corpus comprises a comprehensive overview of a legal case, a specific inquiry associated with it, and a potential argument in support of a solution, supplemented with an in-depth rationale elucidating the applicability of the argument within the given context. Derived from a text designed for legal education purposes, this dataset presents a multifaceted benchmarking task for contemporary legal language models. Our manuscript delineates the approach we adopted for participation in this competition. Specifically, we detail the use of a Mistral 7B model to answer the questions provided. Our only and best submission reaches an F1-score equal to 0.5597 and an Accuracy of 0.5714, outperforming the task's baseline.

\end{abstract}

\section{Introduction}

The content of the Task 5 hosted at SemEval-2024 \cite{held-habernal-2024-legalreasoning}, was originally introduced in \cite{bongard2022legal}. 


Asserting a legal argument represents a fundamental proficiency necessary for aspiring legal professionals to acquire. This proficiency demands not only a comprehension of pertinent legal domains but also advanced reasoning skills, including the utilization of analogy-based arguments and the identification of implicit contradictions. Despite recent strides in establishing objective metrics for contemporary natural language processing (NLP) models across diverse facets of legal language comprehension, the absence of a sophisticated task addressing argumentative reasoning within legal contexts persists.

In this article, is discussed a novel task alongside a corresponding benchmark dataset. The introduction of a genuinely challenging task, sourced from legal educational resources, will serve to elucidate strengths and weaknesses inherent in contemporary legal transformer models, including but not limited to Legal-BERT \citep{chalkidis-etal-2020-legal}. Specifically, at the SemEval-2024 Task 5 is unveiled a novel, openly accessible legal dataset tailored for the binary text classification of issues within U.S. civil procedure. The primary objective is to ascertain whether a proposed solution to a given inquiry is deemed accurate or erroneous. The corpus draws inspiration from "The Glannon Guide To Civil Procedure" authored by Joseph Glannon \cite{glannon2023glannon}, which caters to law students by offering a comprehensive examination of fundamental U.S. civil procedure topics, inclusive of multiple-choice queries designed to assess reader comprehension.

Through the inception of this freshly minted corpus, the intent extends to scrutinizing the efficacy of various methodological approaches while establishing performance benchmarks.

To address these objectives, there is an ongoing demand for automated tools capable of extracting and categorizing data, facilitating the classification with recent NLP models. Recent advancements in the area of the machine and deep learning architectures have spurred heightened interest in Natural Language Processing (NLP). Substantial endeavours have been directed towards devising techniques for the automated identification and categorization of textual content accessible on the internet today. In the literature, to perform text classification tasks, several strategies have already been proposed. In the last fifteen years, some of the most successful strategies have been based on SVM \cite{colas2006comparison,croce2022svm}, on Convolutional Neural Network (CNN) \cite{kim2014convolutional,siino2021detection}, on Graph Neural Network (GNN) \cite{lomonaco2022courage}, on ensemble models \cite{miri2022ensemble,siino2022t100} and, recently, on Transformers \cite{vaswani2017attention, siino2022mcrock}.


Participants in SemEval-2024 Task 5 were tasked as follows. The task at hand involves evaluating the accuracy of an answer candidate provided in response to a question, accompanied by a brief introductory passage pertaining to the subject of the question. The objective is to ascertain whether the candidate answer is indeed incorrect or correct. To face with the task, we propose a Transformer-based approach which made use of Mistral 7B \cite{jiang2023mistral}. We used the model in a zero-shot setup described in the rest of this paper. Specifically, we prompted the latest pre-trained version of Mistral with each sample in the dataset. Specifically, we provided a \textit{candidate answer} to a \textit{question}, asking the model if the answer to the legal question was correct or not. The model replied with a yes or no, eventually providing some further explanation.


The subsequent sections of this work are structured as follows: Section \ref{sec:background} offers background information on Task 5, held at SemEval-2024. In Section \ref{sec:system_overview}, we outline the approach introduced in this study. Section \ref{sec:experimental_setup} delves into the specifics of the experimental setup employed to reproduce our findings. The outcomes of the official task and relevant discussions are presented in Section \ref{sec:results}. Finally, Section \ref{sec:conclusion} concludes our study and suggests avenues for future research.

We make all the code publicly available and reusable on GitHub\footnote{\raggedright\url{https://github.com/marco-siino/SemEval2024/}}.

\section{Background}
\label{sec:background}

For the Task 5 at SemEval-2024 is proposed a legal corpus, publicly accessible for binary text classification tasks focusing on issues within U.S. civil procedure. The primary objective is to determine the correctness of solutions provided in response to specific questions. This corpus draws its content from "The Glannon Guide To Civil Procedure" authored by Joseph Glannon \cite{glannon2023glannon}, tailored for law students. The book encompasses fundamental U.S. civil procedure topics and includes multiple-choice questions aimed at evaluating reader comprehension.

Through collaboration with the author and publisher, task organizers secured permission to utilize the content of "The Glannon Guide To Civil Procedure" for constructing this dataset, which is freely available to the research community. The book comprises 25 chapters, each containing multiple-choice questions pertaining to a particular topic, prefaced by an introduction. Every question is followed by 3 to 5 answer candidates, among which one is deemed correct. These answer candidates serve as hypotheses, necessitating an examination of their respective prerequisites for accuracy. The correctness or incorrectness of an answer is subsequently expounded upon in the accompanying analysis.

The dataset construction process involved automated parsing of the book's content, leveraging its structured format to extract individual components of each instance (i.e., introduction, question, answers, and analysis). Additional parsing rules were employed to detect anomalies in the structure, such as instances where the same introduction was shared across multiple questions. However, certain sections of the book required manual extraction, particularly regarding the correctness of answer candidates, as this information was typically embedded within the free-text analysis section. The analysis segments were organized to address each answer candidate separately, classifying them as true or false. To achieve this, the organizers adopted a strategy of isolating the relevant aspects for each answer, despite the absence of explicit keywords or structural indicators guiding the segmentation process. Despite efforts to maintain consistency, some structural inconsistencies were noted throughout the dataset.

Two samples from provided datasets are available online\footnote{\url{https://github.com/trusthlt/semeval24}} and reported in the Table \ref{tab:samples} in the Appendix section\ref{sec:appendix}. In this case, the two samples contain the same introduction and the same question while providing different answers. Given the Introduction and the Question, the first answer (first row) is wrong, while the second one (second row) is correct.

The organizers adhere to the schedule for SemEval24, which means the following dates:

\begin{itemize}
    \item Tasks announced (with sample data available): 17 July 2023
    \item Training data ready 4 September 2023
    \item Evaluation start 10 January 2024
    \item Evaluation end by by 31 January 2024
    \item Paper submission due 19 February 2024
    \item Notification to authors 18 March 2024
    \item Camera ready due 01 April 2024
    \item SemEval workshop: June 16–21, 2024 (co-located with NAACL 2024 in Mexico City, Mexico)
\end{itemize}



\section{System Overview}
\label{sec:system_overview}


Even if it has already been proved that the Transformers are not necessarily the best option for any text classification task \cite{siino2022fake}, depending on the goal, some strategies like domain-specific fine-tuning \cite{sun2019fine, van2023vietnamese}, or data augmentation \cite{lomonaco2023text,mangione2022improving,siino2024backtranslate} can be beneficial for the considered task.

So far, several Large Language Models (LLMs) have proved to be able to address a plethora of different NLP tasks. For example, in the recent literature, there has been mention of LLaMA, as presented by \cite{touvron2023llama}. LLaMA stands out as a collection of publicly available Large Language Models (LLMs) that rival the capabilities of closed-source counterparts like GPT-3.

However, to address the Task 5 hosted at SemEval-2024 we made use of a zero-shot learning approach \cite{Chen202312844,Wahidur202410146}, making use of Mistral 7B \cite{jiang2023mistral}. Mistral 7B, a language model boasting 7 billion parameters, is engineered to excel in both performance and efficiency. In comparison to the leading open 13B model (Llama 2), Mistral 7B demonstrates superior performance across all assessed benchmarks. Moreover, it outperforms the leading publicly available 34B model (LLaMA 1) across various tasks involving code generation, mathematical operations, and reasoning. The model capitalizes on grouped-query attention (GQA) to expedite inference, complemented by sliding window attention (SWA) to effectively process sequences of varying lengths while minimizing inference costs. Additionally, a fine-tuned variant, Mistral 7B – Instruct, is tailored for adhering to instructions. This version, outperforms Llama 2 13B – chat model across both automated and human benchmarks.

The introduction of Mistral 7B Instruct underscores the ease with which the base model can be fine-tuned to achieve notable performance enhancements. Notably, this variant lacks any moderation mechanisms. 

Our approach is few-shot \cite{littenberg2022comparing} and make use of the above-mentioned Mistral 7B. More specifically, given the task hosted at SemEval-2024, we asked the model: \textit{"Is the Answer to the Question above True or False? Answer using ONLY True or False:"}. To this request, the model replied with one or more words - usually starting with a \textit{true} or \textit{false} - that we parsed to extract one of the two labels (i.e., 0 for false and 1 for true). For example, given the introduction: 

\begin{quote}
    \textit{"Defendant in denial. Cardozo is in an accident on Main Street with two other cars, driven by Hooper and Lopes. Cardozo brings a suit in federal court against Hooper and Lopes for his damages. Paragraph 21 of Cardozo’s complaint alleges that Hooper had signaled before he turned onto Main Street. The police report on the accident states that, according to a bystander, Hooper had signaled before turning onto Main Street. Lopes, who was coming from Hooper’s left, had no view of the right side of Hooper’s car, and did not see whether he signaled or not. At the time an answer is due, Lopes’s counsel has seen the police report, but has not yet been able to locate other witnesses to obtain their testimony. The most appropriate response for Lopes to Paragraph 21 of Cardozo’s complaint would be to."} 
\end{quote}


The answer: 

\begin{quote}
\textit{"state that he is without sufficient information to form a belief about the truth of the allegation."} 
\end{quote}

And our question:

\begin{quote}
    \textit{Is the Answer to the Question above True or False? Answer using ONLY True or False:} 
\end{quote}


The model replied with: 

\begin{quote}
\textit{true. lopes' answer could state that he lacks sufficient information to admit} 
\end{quote}

that we mapped into the binary label \textit{1} corresponding to \textit{true}.

We did not find any inconsistency in the outputs generated by Mistral along all the provided prompts. Specifically, we did not notice any variation in the behaviours of the model at different times of prompting. This leads us to the conclusion that given always the same input context (i.e., few-shot samples) during the prompt, the output provided is always consistent disregarding the time and the previous prompts provided. Finally, we collected all the predictions provided on the test set to into a JSON file with the required format to submit our predictions. 


As noted in the recent study by \cite{siino2024text}, the contribution of preprocessing for text classification tasks is generally not impactful when using Transformers. More specifically, the best combination of preprocessing strategies is not very different from doing no preprocessing at all in the case of Transformers. For these reasons, and to keep our system highly fast and computationally light, we have not performed any preprocessing on the text.


\section{Experimental Setup}
\label{sec:experimental_setup}

We implemented our model on Google Colab. The library we used come from HuggingFace and as already mentioned is Mistral 7B\footnote{\url{https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF}}. We employed the v0.2 iteration of Mistral 7B, which represents an enhanced version of the Mistral-7B-Instruct-v0.1 model. To harness the capabilities of instruction fine-tuning, prompts must be enclosed within [INST] and [/INST] tokens. Additionally, the initial instruction should commence with a sentence identifier. The next instructions should not. The assistant generation will be ended by the end-of-sentence token ID. We also imported the Llama library \cite{touvron2023llama} from \textit{llama\_cpp}. The library is fully described on GitHub\footnote{\url{https://github.com/ggerganov/llama.cpp}}. The dataset provided for all the phases are available on the official competition page. We did not perform any additional fine-tuning on the model. To run the experiment, a T4 GPU from Google has been used. After the generation of predictions, we exported the results on the format required by the organizers. As already mentioned, all of our code is available on GitHub.


\section{Results}
\label{sec:results}


Given the binary nature of the classification task, the organizers proposed F1 score and Accuracy as the two evaluation metrics to be considered for the final ranking. The F1 score is defined in the Equation \ref{eq:f1}. Where TP stands for the number of correctly predicted right answers, FP stands for the number of wrongly predicted right answers, and FN stands for the right answers wrongly predicted as wrong answers.


\begin{equation}
\label{eq:f1}
F1 = \frac{2*Precision*Recall}{Precision+Recall} 
\end{equation}

Given the previous definitions, the accuracy is defined as stated in the Equation \ref{eq:accuracy}.


\begin{equation}
\label{eq:accuracy}
Accuracy = \frac{TP+TN}{TP+TN+FP+FN}
\end{equation}



In Table \ref{tab:mistral_results}, we present the outcomes derived from our methodology. They are the same results publicly availble on the official final ranking shown on the official task page\footnote{\url{https://github.com/trusthlt/semeval24}} and on CodaLab\footnote{\url{https://codalab.lisn.upsaclay.fr/competitions/14817}}. 

In the Table \ref{tab:competition_results}, the results obtained by the first three teams and by the last one, as showed on the official task page, are reported. Compared to the best performing models, our simple approach exhibits some room for improvements. However, it is worth notice that required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by Google Colab. Finally, the proposed approach is able to outperform the baseline provided.


\begin{center}
\begin{table}
\begin{center}
\begin{tabular}{ ccc } 
    \hline
  & F1 & Accuracy \\ 
  \hline
  Mistral 7B & 0.5597 & 0.5714  \\ 
  \hline
\end{tabular}
\end{center}
\caption{\label{tab:mistral_results}The method's performance on the test set. In the table, the results obtained and shown on the official GitHub page are reported.}
\end{table}
\end{center}



\begin{center}
\begin{table*}[ht]
\begin{center}
\begin{tabular}{ ccc } 
    \hline
   TEAM NAME & F1 & Accuracy \\ 
  \hline
  HW-TSC (1) & 0.8231 & 0.8673 \\ 
  \hline
  PoliToHFI (2) & 0.7747 & 0.8265 \\ 
   \hline
  SU-FMI (3) & 0.7728 & 0.8367 \\
  \hline
  lena.held (21) & 0.4269 & 0.7449 \\ 
  \hline
\end{tabular}
\end{center}
\caption{\label{tab:competition_results}Comparing performance on the test set. In the table are shown the results obtained by the first three teams and by the last one.  In parentheses is reported the position in the official final ranking.}
\end{table*}
\end{center}


\section{Conclusion}
\label{sec:conclusion}


This paper presents the application of a Mistral 7B-model for addressing the Task 5 at SemEval-2024. For our submission, we decided to follow a zero-shot learning approach, employing as-is, an in-domain pre-trained Transformer. After several experiments, we found beneficial to build a prompt containing the question for the model. Then we provide as a prompt: the introduction, the question and an answer candidate. The model is asked to decide whether the candidate answer is correct or not. The task is challenging, and there is still opportunity for improvement, as can be noted looking at the final ranking. Possible alternative approaches include utilizing the few-shot capabilities or also the use of other models like GPT and T5, eventually using further data, or directly integrating other samples from the training and from the development sets. Further improvements could be obtained with a fine-tuning and modelling the problem as a text classification task. Furthermore, given the interesting results recently provided on a plethora of tasks, also other few-shot learning \cite{wang2023few, maia2024transformers, siino2023profiling, meng2024hierarchical} or data augmentation strategies \cite{muftie2023indobert, tapia2020data, siino2023xlnet} could be employed to improve the results. Looking at the final ranking, our simple approach exhibits some room for improvements. However, it is worth notice that required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by Google Colab. 
\section*{Acknowledgments}
We extend our gratitude to the anonymous reviewers for their insightful comments and valuable suggestions, which have significantly enhanced the clarity and presentation of this paper.

% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}

\appendix

\section{Appendix}
\label{sec:appendix}
As stated in the background section, in this appendix are shown two samples from the provided datasets. The two samples in the Table \ref{tab:samples} give an example of a wrong answer candidate (first row in the table) and an example of a correct answer candidate (second row in the table). 

\begin{center}
\begin{table*}[ht]
\begin{center}
\begin{tabular}{ |p{5.5cm}|p{5cm}|p{3cm}|p{0.8cm}| } 
    \hline
   Introduction & Question & Answer Candidate & Label \\ 
  \hline
  \textit{"My students always get confused about the relationship between removal to federal court and personal jurisdiction. Suppose that a defendant is sued in Arizona and believes that she is not subject to personal jurisdiction there. Naturally, she should object to personal jurisdiction. [...] But generally the scope of personal jurisdiction in the federal court will be the same as that of the state court, because the Federal Rules require the federal court in most cases to conform to state limits on personal jurisdiction. Fed. R. Civ. P. 4(k)(1)(A). I’ve stumped a multitude of students on this point. Consider the following two cases to clarify the point."} & \textit{"7. A switch in time. Yasuda, from Oregon, sues Boyle, from Idaho, on a state law unfair competition claim, seeking \$250,000 in damages. He sues in state court in Oregon. Ten days later (before an answer is due in state court), Boyle files a notice of removal in federal court. Five days after removing, Boyle answers the complaint, including in her answer an objection to personal jurisdiction. Boyle’s objection to personal jurisdiction is"} & not waived by removal, but will be denied because the federal courts have power to exercise broader personal jurisdiction than the state courts. & 0 \\ 
  \hline
  \textit{"My students always get confused about the relationship between removal to federal court and personal jurisdiction. Suppose that a defendant is sued in Arizona and believes that she is not subject to personal jurisdiction there. Naturally, she should object to personal jurisdiction. [...] But generally the scope of personal jurisdiction in the federal court will be the same as that of the state court, because the Federal Rules require the federal court in most cases to conform to state limits on personal jurisdiction. Fed. R. Civ. P. 4(k)(1)(A). I’ve stumped a multitude of students on this point. Consider the following two cases to clarify the point."} & \textit{"7. A switch in time. Yasuda, from Oregon, sues Boyle, from Idaho, on a state law unfair competition claim, seeking \$250,000 in damages. He sues in state court in Oregon. Ten days later (before an answer is due in state court), Boyle files a notice of removal in federal court. Five days after removing, Boyle answers the complaint, including in her answer an objection to personal jurisdiction. Boyle’s objection to personal jurisdiction is"} & not waived by removal. The court should dismiss if there is no personal jurisdiction over Boyle in Oregon, even though the case was properly removed.
 & 1 \\ 
  \hline
\end{tabular}
\end{center}
\caption{\label{tab:samples}Two different samples from the official dataset are provided. Together with the introduction, a question and a candidate answer the label is provided (i.e., 0 if the answer is incorrect, 1 if the answer is correct) }
\end{table*}
\end{center}



\end{document}
