\pdfoutput=1
\documentclass[11pt]{article}
\usepackage[T1]{fontenc}
\usepackage{url}
\usepackage{hyperref}
\hypersetup{  
    colorlinks=true,  
    linkcolor=blue,  
    anchorcolor=blue,  
    citecolor=blue  
}
\usepackage{acl}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{times}
\usepackage{latexsym}
\usepackage{booktabs}
\usepackage[utf8]{inputenc}
\usepackage{microtype}
\usepackage{inconsolata}
\title{\textbf{\textbf{YNU-HPCC at S}}\textbf{\textbf{em}}\textbf{\textbf{Eval-2024 Task 5: Regularized Legal-BERT for Legal Argument Reasoning Task in Civil Procedure}}}
\author{Peng Shi, Jin Wang and Xuejie Zhang \\
  School of Information Science and Engineering \\
  Yunnan University \\
  Kunming, China \\
\texttt{Shipeng1@stu.ynu.edu.cn}, \texttt{\{wangjin, xjzhang\}@ynu.edu.cn}}
\begin{document}
\maketitle
\begin{abstract}
This paper describes the submission of team YNU-HPCC to SemEval-2024 for Task 5: The Legal Argument Reasoning Task in Civil Procedure. The task asks candidates the topic, questions, and answers, classifying whether a given candidate's answer is correct (True) or incorrect (False). To make a sound judgment, we propose a system. This system is based on fine-tuning the Legal-BERT model that specializes in solving legal problems. Meanwhile, Regularized Dropout (R-Drop) and focal Loss were used in the model. R-Drop is used for data augmentation, and focal loss addresses data imbalances. Our system achieved relatively good results on the competition's official leaderboard. The code of this paper is available at \url{https://github.com/YNU-PengShi/SemEval-2024-Task5}. 
\end{abstract}
\section{Introduction}
The task can be formulated as follows: given an introduction to the topic, a question, and an answer candidate, classify if the given candidate is correct (True) or incorrect (False) \cite{bongard-etal-2022-legal}. This task has two main difficulties: 1) The text length of the topic and question is much larger than 512 tokens. 2) The number of positive and negative samples in the data varies widely. 
\par Initially, the online system represented the first attempt to utilize computational methods for addressing legal conundrums \cite{VALENTE19991079}. Despite notable advancements in recent years, which have seen a concerted effort to establish objective benchmarks for natural language processing models in the domain of legal language comprehension \cite{chalkidis-etal-2022-lexglue}, a lack remains in the realm of complex tasks involving argumentative reasoning within legal contexts. However, Legal-BERT has emerged as a forerunner in this domain, demonstrating compelling performance \cite{chalkidis-etal-2020-legal}.
\par This paper proposes a model based on Legal-BERT. In processing tasks, we used sliding window simple (SWS) and sliding window complex (SWC) to process the original data and solved the problem of the token count of the original data being much larger than 512. In the subsequent process, we found that there was a significant imbalance in the dataset that resulted in the return of the most common label in the training set (in this case, 0). We added R-Drop \cite{wu2021r} to the model to address this issue and changed the loss function from cross entropy to focal loss \cite{lin2017focal}. In the end, we achieved a good result. The best submission for the test set has achieved 0.6166 and ranked 9th in this task.
\par The remainder of this paper is organized as follows. Section \hyperref[pra:2]{2} describes the model and method used in our system, section \hyperref[pra:3]{3} discusses the results of the experiments, and finally, the conclusions are drawn in section \hyperref[pra:4]{4}.
\section{System Description}
\label{pra:2}
This section delves into the intricate design of the proposed model's architecture. The architecture comprises multiple essential components, namely the text cutting, the tokenizer, the pre-trained Legal-BERT model, the output layer, and the methods. Figure \hyperref[fig:The structure of system]{1} illustrates the comprehensive system model that we have devised.
\begin{figure}[t!]
    \centering
    \label{fig:The structure of system}
    \includegraphics[width=\linewidth,height=0.6\textwidth]{Figure1_00.png}
    \caption{The structure of system}
    \end{figure} 
\subsection{Text Preprocessing}
\textbf{Sliding Window Simple (SWS).} The process involves dividing the combined question and introduction into discrete segments or chunks. These chunks are then submitted to a classification algorithm, which assigns a category or label to each segment based on its content. Once the classification is complete, the system calculates the average predicted output for all the chunks. This average serves as a comprehensive summary or representation of the combined text, capturing the key themes and characteristics. It's a method that leverages machine learning techniques to distill the essence of a complex textual input into a single numerical value, which can be helpful in various applications such as summarization, sentiment analysis, and information retrieval.\vspace{0.2cm}\\
\textbf{Sliding Window Complex (SWC).} In this sophisticated text processing workflow, the initial step decomposes the introductory text into discrete segments or chunks. Each chunk is meticulously constructed to include the complete question, flanked by the introduction's segments to provide context. This approach ensures that each chunk is a self-contained unit that retains the connectivity between the question and the supporting information in the introduction \cite{koay-etal-2021-sliding}.
\par Subsequently, these meticulously crafted chunks are subjected to a comprehensive classification process. This process employs advanced machine learning algorithms to analyze the content of each chunk and assign it to one or more predefined categories or labels. The classification is nuanced and context-aware, considering the intricate details and subtle nuances present in the text \cite{KONG2022107872}.
\par The system employs a statistical aggregation technique to calculate the average of the predicted outputs for all the chunks. This average is a weighted sum of the individual predictions, giving more weight to chunks deemed more critical or relevant based on the specific application context.
\par The resulting average is a valuable metric that encapsulates the collective predictions of the model for the given question and introduction. It provides a robust summary of the model's understanding of the text, offering insights into the key themes and conclusions the model has extracted from the input. This average output can be used for various applications, such as generating summaries, making predictions, or informing decision-making processes.
\subsection{Tokenizer}
In many natural language processing (NLP) tasks, the original text must be processed into digital data before it can be processed by computer. Thus, the tokenizer was applied to divide the text into words and convert it into unique coding. Given a training data ${\cal D} = \{ {X^{(m)}},{y^{(m)}}\} _{m = 1}^M$, ${X^{({m})}}$ is the processed input text. ${y^{(m)}}$is the corresponding ground-true label, the Bert tokenizer is applied to transform ${X^{(m)}}$ as,
\begin{equation}
\resizebox{0.85\hsize}{!}{$\begin{aligned}
X = [CLS]{x_1}{x_2}{x_3}...{x_n}[SEP]{y_1}{y_2}...{y_{{m}}}[SEP]
\end{aligned}$}
\end{equation}
where $x$ and $y$ represent tokens, $n$ and $m$ represent the length of the first and second sentences, $[CLS]$ special mark indicates the beginning of the text sequence, $[SEP]$ indicates the separator between text sequences, respectively. 
\subsection{Legal-BERT Model}
Legal-BERT is a specialized variant of the BERT model tailored for the legal domain, leveraging a corpus of legal text to facilitate advancements in legal natural language processing research, computational law, and legal technology applications \cite{chen-etal-2023-ynu}. This model inherits the parameter weights from BERT-Base, ensuring a solid foundation for legal-specific tasks. In our study, we employed the pre-trained Legal-BERT model, built upon the Transformer library \footnote{\url{https://huggingface.co/nlpaueb/legal-bert-base-uncased}}, to handle the complexities of legal language. The architecture of Legal-BERT mirrors that of the original BERT model, comprising an essential components: the Transformer encoder block \cite{vaswani2017attention}. These blocks work to capture legal text's intricate patterns and nuances. The model configuration used in our experiment features 12 layers, 768 dimensions, 12 self-attention heads, and a total of 109 million parameters. This configuration balances model complexity and computational efficiency, enabling us to tackle various legal NLP challenges effectively.\vspace{0.2cm}\\
\textbf{Encoder block.} Firstly, Legal-BERT performs the embedding operation after receiving the processed raw data. Through the above processing, we obtained token embedding, segment embedding, and position embedding \cite{zhang2021personalized}, followed by a series of operations to obtain \textbf{H}, as follows.
\begin{equation}
    {\bf{H}} = {\mathop{\rm Enc}\nolimits} (X;\theta )
\end{equation}
where ${\bf{H}} \in {{\mathbb{R}}^d}$ is the logits with a dimensionality of 768.
\subsection{Output Layer}
The BERT model has two major pretraining tasks: mask language model (MLM) and next sentence prediction (NSP), and the text implication task usually uses the NSP method to predict, that is, use the hidden layer representation of $[CLS]$ bits to predict the text classification \cite{ma-etal-2021-ynu}. In our proposed model, the output of the model is first to use a softmax function and then perform argmax on the results after softmax to obtain $\hat y$,
\begin{equation}
    \hat y = {\mathop{\rm argmax}\nolimits} ({\mathop{\rm softmax}\nolimits} ({W^o}{\bf{H}} + {h^o}))
\end{equation}
The training objective is to optimize the focal loss between the true and predicted labels,
\begin{equation}
\resizebox{0.85\hsize}{!}{$\begin{aligned}
  {{\cal L}_{FL}} = \left\{ {\begin{array}{*{20}{c}}
{ - {{(1 - \hat y_{}^{(m)})}^\gamma }\log (\hat y_{}^{(m)})\quad if\ {y^{(m)}} = 1}\\
{ - \hat y{{_{}^{(m)}}^\gamma }\log (1 - \hat y_{}^{(m)})\quad if\ {y^{(m)}} = 0}
\end{array}} \right.  
\end{aligned}$}
\end{equation}
where ${W^o} \in {{\mathbb{R}}^d}$ represents the weight of the fully connected layer, $h^o$ represents the offset of the fully connected layer, ${\bf{H}} \in {{\mathbb{R}}^d}$ is the output representation of $[CLS]$ token in the L-th layer, $\gamma$ is used to control the weight of difficult-to-classify samples, ${y^{(m)}}$ are respectively the true label,  $\hat y_{}^{(m)}$ are respectively the probability distribution of prediction.
\subsection{Regularized Dropout (R-Drop)}
To solve the problem of highly imbalanced data, R-Drop is added to the output layer of Legal-BERT. As shown in Figure \hyperref[fig:The structure of R-Drop]{2}, the same input can obtain two logits, ${\textbf{H}_1}$ and ${\textbf{H}_2}$, respectively, during the R-Drop process. Therefore, the model will output two predicted values $\hat y_{}^{(1)}$ and $\hat y_{}^{(2)}$, as follows.
\begin{equation}
    \hat y_{}^{(1)} = {\mathop{\rm argmax}\nolimits} ({\mathop{\rm softmax}\nolimits} ({W^o}{{\bf{H}}_1} + {h^o}))
\end{equation}
\begin{equation}
    \hat y_{}^{(2)} = {\mathop{\rm argmax}\nolimits} ({\mathop{\rm softmax}\nolimits} ({W^o}{{\bf{H}}_2} + {h^o}))
\end{equation}
R-Drop uses a symmetrical Kullback-Leibler (KL) divergence to constrain $\hat y_{}^{(1)}$ and $\hat y_{}^{(2)}$, as follows.
\begin{equation}
\resizebox{0.85\hsize}{!}{$\begin{aligned}
    {{\cal L}^i}_{KL} = \frac{1}{2}(({D_{KL}}(\hat y_{}^{(1)}||\hat y_{}^{(2)}) + {D_{KL}}(\hat y_{}^{(2)}||\hat y_{}^{(1)}))
\end{aligned}$}
\end{equation}
Finally, the model will calculate the loss of two predicted values $\hat y_{}^{(1)}$ and $\hat y_{}^{(2)}$ using focal loss, as follows.
\begin{equation}
\resizebox{0.85\hsize}{!}{$\begin{aligned}
    {\cal L}_{FL}^1 = \left\{ {\begin{array}{*{20}{c}}
{ - {{(1 - \hat y_{}^{(1)})}^\gamma }\log (\hat y_{}^{(1)})\quad if\ {y^{(1)}} = 1}\\
{ - \hat y{{_{}^{(1)}}^\gamma }\log (1 - \hat y_{}^{(1)})\quad if\ {y^{(1)}} = 0}
\end{array}} \right.
\end{aligned}$}
\end{equation}

\begin{equation}
\resizebox{0.85\hsize}{!}{$\begin{aligned}
    {\cal L}_{FL}^1 = \left\{ {\begin{array}{*{20}{c}}
{ - {{(1 - \hat y_{}^{(2)})}^\gamma }\log (\hat y_{}^{(2)})\quad if\ {y^{(2)}} = 1}\\
{ - \hat y{{_{}^{(2)}}^\gamma }\log (1 - \hat y_{}^{(2)})\quad if\  {y^{(2)}} = 0}
\end{array}} \right.
\end{aligned}$}
\end{equation}
The training loss function for Legal-BERT is as follows.
\begin{equation}
    {{\cal L}_i} = {\cal L}_{FL}^1 + {\cal L}_{FL}^2 + {\cal L}_{KL}^i
\end{equation}
\begin{figure}[t!]
\label{fig:The structure of R-Drop}
    \centering
    \includegraphics[width=\linewidth,scale=1.00,height=0.3\textwidth]{Figure2_00.png}
    \caption{The structure of R-Drop}
    \end{figure}
\section{Experimental Result}
\label{pra:3}
\textbf{Datasets.} The Legal Argument Reasoning Task in Civil Procedure shared task data set is composed of three CSV files: the size of the training set train.csv sorted by expert comments is 666, the size of the developing set dev.csv is 84, the size of test set test.csv is 98. The data part of the train and dev set mainly includes idx, question, answer, label, analysis, complete analysis, and explanation. The data part of the test set mainly includes idx, question, answer, and explanation. Idx is used to represent the number of each sample. The question is made in the context of the content of the explanation. The answer is a candidate answer in the sample. Label indicates whether the question and candidate answer match, 0 for mismatch, and 1 for matching. Analysis and complete analysis are used for experimenters to understand why the label is 0 or 1. Explanation is used to indicate the subject of the sample to which it belongs.\vspace{0.2cm}\\
\begin{figure}[t!]
    \label{fig: Examples of different models on the dev set}
    \centering
    \includegraphics[width=\linewidth,height=0.6\textwidth]{Figure3_00.png}
    \caption{Examples of different models on the dev set}
\end{figure}\textbf{Evaluation Metrics.} The Legal Argument Reasoning Task in Civil Procedure shared tasks are evaluated using the standard evaluation indicators, including Macro $F_1$-\textit{score} and Accuracy. The submissions of all teams are ranked according to the $F_1$-\textit{score}. The metrics will be calculated as follows.
\begin{equation}
\resizebox{0.85\hsize}{!}{$\begin{aligned}
Precision = \frac{{true\; positives}}{{true\; positives + false\; positives}}
\end{aligned}$}
\end{equation}
\begin{equation}
\resizebox{0.85\hsize}{!}{$\begin{aligned}
    Recall = \frac{{true\; positives}}{{true\; positives + false\; negatives}}
\end{aligned}$}
\end{equation}
\begin{equation}
\resizebox{0.85\hsize}{!}{$\begin{aligned}
    {F_1}\mbox{-}{score} = \frac{{2 \times Precision \times Recall}}{{Precision + Recall}}
    \end{aligned}$}
\end{equation}
\\
\textbf{Implementation Details.} Initially, explanation and question are concatenated when processing data. The BERT \cite{devlin2018bert} is used as the first model to solve this task. However, without any treatment, the predicted value of the BERT is all 0, and the effect is not ideal. Next, we used the larger models RoBERTa \cite{liu2019roberta} and DeBERTa \cite{he2020deberta}, but the predictions and $F1\_{\text{scores}}$ were identical to BERT. Due to the extreme data imbalance, we found that the cross-entropy loss function could not calculate the loss correctly. Therefore, we changed the loss function for BERT, RoBERTa, and DeBERTa to focal loss and dice loss. The results show that modifying the loss function can slightly improve the score, but the effect is not ideal. To solve the problem of extreme data imbalance further, we change their loss functions to focal loss and dice loss \cite{li-etal-2020-dice} based on supervised contrastive learning (SCL) \cite{NEURIPS2020_d89a66c7} and R-Drop. The results show that the combination of pairs can effectively solve the problem of extreme data imbalance, and the score has also been significantly improved. During the experiment, we found that due to the large number of proprietary legal terms in the data text, the above model could not fully segment professional vocabulary using the corresponding tokenizer. Therefore, we believe that the Legal-BERT is the most suitable choice. As expected, Legal-BERT has achieved good results in adding R-Drop and focal Loss technologies, as shown in Figure \hyperref[fig: Examples of different models on the dev set]{3}.
\vspace{0.2cm}\\
\textbf{Hyper-parameters Fine-tuning.} We adjusted different learning rates and epochs to adapt to different models to achieve the expected results. Legal-BERT is better than BERT regardless of the learning rate, as shown in Figure \hyperref[fig:The performance of different learning rates on the f1]{4}. 
\begin{figure}[t!]
    \label{fig:The performance of different learning rates on the f1}
    \centering
    \includegraphics[width=\linewidth,height=0.3\textwidth]{Figure4_00.png}
    \caption{The performance of different learning rates on the $F_1$-\textit{score}}
    \end{figure}
\begin{figure}[t!]
    \label{fig: The performance of different batch sizes on 
    the f1}
    \centering
    \includegraphics[width=\linewidth,height=0.3\textwidth]{Figure5_00.png}
    \caption{The performance of different batch sizes on the 
    $F_1$-\textit{score}}
    \end{figure}
The optimal $F_1$-\textit{score} was found at 4 with the batch size constantly changing, as shown in Figure \hyperref[fig: The performance of different batch sizes on the f1]{5}. 
We set the best parameters in the final submitted results: warmup steps are 10, weight decay is 0.01, the learning rate is 3e-5, train batch size is 4, and epoch is 100.\vspace{0.2cm}\\
\textbf{Comparative Results and Discussion.} The test is first carried out on the development set, whose size is 84. Facing the different predicted results of other models and Legal-BERT, it is clear that Legal-BERT performs better. Regardless of the model, as long as the loss function is cross entropy, the final predicted value will be 0. Both dice loss and focal loss can solve the problem of imbalance in data, but focal loss is more effective. When SCL and R-Drop were introduced, R-Drop achieved significantly better results. Legal-BERT can deal with legal vocabulary more thoroughly than other models. Overall, Legal-BERT+R-Drop+focal Loss is the best combination obtained after experiments.
The $F_1$-\textit{score} obtained from the experiments of several models and methods is summarized in Table \hyperref[tab:1]{1}, Table \hyperref[tab:2]{2}, and Table \hyperref[tab:3]{3}, and the result of the best submission is shown in Table \hyperref[tab:4]{4}. Although the sliding window approach helps alleviate the token limitations of Legal-BERT, models specifically designed to handle longer documents, such as Longformer \cite{beltagy2020longformer} or Big Bird \cite{zaheer2020big}, might offer superior efficiency. In the future, our team will also use the above model to solve the problem of long text.
\begin{table}[t!]
\normalsize
\label{tab:1}
\centering
\scalebox{0.7}{\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Model} & \textbf{Loss} & \textbf{$F_1$-\textit{score}} & \textbf{Accuracy} \\ \hline
BERT           & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
RoBERTa        & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
DeBERTa        & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
Legal-BERT     & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
BERT           & Focal Loss    & 0.4688            & 0.8095            \\ \hline
RoBERTa        & Focal Loss    & 0.4437            & 0.7976            \\ \hline
DeBERTa        & Focal Loss    & 0.4956            & 0.7976            \\ \hline
Legal-BERT     & Focal Loss    & 0.5599            & 0.6548            \\ \hline
BERT           & Dice Loss     & 0.5468            & 0.6548            \\ \hline
RoBERTa        & Dice Loss     & 0.4830            & 0.7738            \\ \hline
DeBERTa        & Dice Loss     & 0.4830            & 0.7738            \\ \hline
Legal-BERT     & Dice Loss     & 0.4943            & 0.7421            \\ \hline
\end{tabular}}
\caption{models and methods.}
\end{table}

\begin{table}[t!]
\normalsize
\label{tab:2}
\centering
\scalebox{0.7}{\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Model}   & \textbf{Loss} & \textbf{$F_1$-\textit{score}} & \textbf{Accuracy} \\ \hline
BERT + SCL       & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
RoBERTa + SCL    & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
DeBERTa + SCL    & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
Legal-BERT + SCL & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
BERT + SCL       & Focal Loss    & 0.5625            & 0.6428            \\ \hline
RoBERTa + SCL    & Focal Loss    & 0.5460            & 0.8095            \\ \hline
DeBERTa + SCL    & Focal Loss    & 0.4247            & 0.7381            \\ \hline
Legal-BERT + SCL & Focal Loss    & 0.5296            & 0.6706            \\ \hline
BERT + SCL       & Dice Loss     & 0.4892            & 0.7302            \\ \hline
RoBERTa + SCL    & Dice Loss     & 0.4437            & 0.7976            \\ \hline
DeBERTa + SCL    & Dice Loss     & 0.4437            & 0.7976            \\ \hline
Legal-BERT + SCL & Dice Loss     & 0.5299            & 0.6508            \\ \hline
\end{tabular}}
\caption{models and methods.}
\end{table}

\begin{table}[t!]
\normalsize
\label{tab:3}
\centering
\scalebox{0.7}{\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Model}      & \textbf{Loss} & \textbf{$F_1$-\textit{score}} & \textbf{Accuracy} \\ \hline
BERT + R-Drop       & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
RoBERTa + R-Drop    & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
DeBERTa + R-Drop    & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
Legal-BERT + R-Drop & Cross-Entropy & 0.4437            & 0.7976            \\ \hline
BERT + R-Drop       & Focal Loss    & 0.5637            & 0.6746            \\ \hline
RoBERTa + R-Drop    & Focal Loss    & 0.4437            & 0.7976            \\ \hline
DeBERTa + R-Drop    & Focal Loss    & 0.5650            & 0.6510            \\ \hline
Legal-BERT + R-Drop & Focal Loss    & 0.6690            & 0.8210            \\ \hline
BERT + R-Drop       & Dice Loss     & 0.4824            & 0.6310            \\ \hline
RoBERTa + R-Drop    & Dice Loss     & 0.4437            & 0.7976            \\ \hline
DeBERTa + R-Drop    & Dice Loss     & 0.5155            & 0.6310            \\ \hline
Legal-BERT + R-Drop & Dice Loss     & 0.4437            & 0.7976            \\ \hline
\end{tabular}}
\caption{models and methods.}
\end{table}

\begin{table}[t!]
\scriptsize
\label{tab:4}
\centering
\begin{tabular}{|c|c|}
\hline
\textbf{${F_1}$-\textit{score}} & \textbf{Accuracy} \\ \hline
0.6166            & 0.6837            \\ \hline
\end{tabular}
\caption{best submission result.}
\end{table}
\section{Conclusion}
\label{pra:4}
In this research paper, we introduce a system submitted for evaluation in SemEval-2024 Task 5. Leveraging the powerful pre-trained Legal-BERT model as its foundation, our system underwent essential modifications to enhance performance. Specifically, we refined the loss function and incorporated the R-Drop technique to determine the alignment between questions and their corresponding answers accurately. The empirical results obtained from our experiments demonstrate the effectiveness of our proposed system, showcasing its strong performance capabilities. However, when benchmarked against the leading systems in the competition, it becomes evident that there are still notable areas for further improvement. Looking ahead, we are eager to explore the integration of alternative legal-specific models and innovative length text processing strategies. By pursuing these avenues, we aim to achieve even more promising results that can contribute significantly to advancing the field.
\section*{Acknowledgement}
The authors would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos.61966038 and 62266051.
\bibliography{custom,anthology,book}
\bibliographystyle{acl_natbib}
\end{document}
