% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{cgloss4e}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Challenge at SemEval 2024 Task 7: Contrastive Learning Approach on Numeral-Aware Language Generation}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Ali-Zuhnis \\ Address line \\  ... \\ Address line
%         \And
%         Hao-Yun Chuang \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Ali Zhunis \\
  University of Tübingen \\
  Tübingen, Germany \\
%  Affiliation / Address line 2 \\
%  Affiliation / Address line 3 \\
  \texttt{ali.zhunis@student.uni-tuebingen.de} \\\And
  Hao-Yun Chuang \\
  National Chengchi University\\
  Taipei, Taiwan \\
%  Affiliation / Address line 2 \\
%  Affiliation / Address line 3 \\
  \texttt{110555010@nccu.edu.tw} \\}

% ============================================================================
%                             REVIEWER #1
% ============================================================================

% ---------------------------------------------------------------------------
% Reviewer's Scores
% ---------------------------------------------------------------------------
%                    Appropriateness (1-5): 5
%                            Clarity (1-5): 3
%       Originality / Innovativeness (1-5): 2
%            Soundness / Correctness (1-5): 2
%              Meaningful Comparison (1-5): 2
%                       Thoroughness (1-5): 2
%         Impact of Ideas or Results (1-5): 2
%                     Recommendation (1-5): 2
%                Reviewer Confidence (1-5): 4

% Detailed Comments
% ---------------------------------------------------------------------------
% The employed model leverages ChatGPT for generating distorted examples aimed at enhancing the sensitivity of numerical models through contrastive learning. While this approach appears straightforward, it lacks significant innovation, leaving room for improvement. The comparison solely against BART, a limited baseline, raises concerns about the comprehensiveness of the evaluation. Furthermore, the absence of detailed comparisons in numerical accuracy assessment, particularly against the original benchmark (NumHG: A Dataset for Number-Focused Headline Generation), is notable.
% ---------------------------------------------------------------------------


% Questions for Authors
% ---------------------------------------------------------------------------
% Q1: Can we trust ChatGPT's proficiency in distorting headlines reliably? Has a thorough examination of the final results generated by ChatGPT been conducted? It's crucial to consider the potential for errors, especially in numerical transformations.
% Q2: How were samples selected for contrastive loss? Were they exclusively from the distorted headlines, or did they include headlines from other news articles? Clarity on the sampling methodology is essential for understanding the scope and effectiveness of the contrastive learning approach.
% ---------------------------------------------------------------------------



% ============================================================================
%                             REVIEWER #2
% ============================================================================

% ---------------------------------------------------------------------------
% Reviewer's Scores
% ---------------------------------------------------------------------------
%                    Appropriateness (1-5): 5
%                            Clarity (1-5): 4
%       Originality / Innovativeness (1-5): 4
%            Soundness / Correctness (1-5): 5
%              Meaningful Comparison (1-5): 4
%                       Thoroughness (1-5): 4
%         Impact of Ideas or Results (1-5): 4
%                     Recommendation (1-5): 4
%                Reviewer Confidence (1-5): 4

% Detailed Comments
% ---------------------------------------------------------------------------
% Clarity: The paper is well-written.

% Originality / Innovativeness: The paper demonstrates innovative methodologies such as data augmentation and contrastive learning.

% Performance: Despite using smaller models like BART and T5, achieving good scores for a challenging task is commendable.
% ---------------------------------------------------------------------------



% ============================================================================
%                             REVIEWER #3
% ============================================================================

% ---------------------------------------------------------------------------
% Reviewer's Scores
% ---------------------------------------------------------------------------
%                    Appropriateness (1-5): 4
%                            Clarity (1-5): 3
%       Originality / Innovativeness (1-5): 3
%            Soundness / Correctness (1-5): 4
%              Meaningful Comparison (1-5): 3
%                       Thoroughness (1-5): 3
%         Impact of Ideas or Results (1-5): 3
%                     Recommendation (1-5): 4
%                Reviewer Confidence (1-5): 5

% Detailed Comments
% ---------------------------------------------------------------------------
% The introduction section of the paper needs the following modifications.
% Content presented from lines 23 to 29 is not required; instead, the problem statements should be defined.
% Move content from Line number 57 to 68 to the dataset section. Under section 4, create a subsection 4.1 "Dataset". Discuss the details of the dataset. In subsection 4.1, "Dataset" is not in the introduction.
% Move content from Line number 51 to 56, Experimental setup section. Under section 4, create a subsection 4.1, "Experimental Setup". They discuss the details of evaluation metrics and hyperparameters used in experiments.
% In the introduction section, add why this problem is important to solve. Implication of this problem etc. How is this problem different from traditional summarization methods?

% Explain how numeral headline generation is different from traditional summarization methods.

% Move section 3.5 to the Experimental Setup section. Under section 4, create a subsection 4.1, "Experimental Setup", and discuss the details of evaluation metrics and details of hyperparameters used in experiments.

% Move section 3.6 to the Experimental Setup section. Under section 4, create a subsection 4.1, "Experimental Setup". Here, discuss the details of evaluation metrics and details of hyperparameters used in experiments.

% I could not find any error analysis in the paper. Kindly perform an error analysis of the response of the proposed model. Add the conclusion and discuss the error analysis in a separate section, "Error Analysis" . Also, present and discuss some samples where the proposed methods fail. Such error analysis is beneficial for subsequent research on the same topic.
% ---------------------------------------------------------------------------

% - Your approach of creating positive/negative examples feels
%  rather circular. Your motivation for the work says that you do
%  this because the LLMs are bad at it, but then, you use an LLM
%  to generate your training instances. I think a more precise
%  approach (e.g., hand-annotated or based on some rule-based
%  system) would probably be better.

% - In quite a few places, you discuss "personal experience". These
%  should be excluded any research paper. In general, any detail
%  (e.g., the date of test set release) that does not affect the
%  work, should be left out.

% - I also think that your approach needs a bit better description.
%  From your description, I still have difficulties, how you made
%  use of the CL. A reader even less familiar with you did, will
%  probably be able to understand less.

\begin{document}
\maketitle
\begin{abstract}
Although Large Language Model (LLM) excels on generating headline on ROUGE evaluation, it still fails to reason number and generate news article headline with an accurate number. Attending SemEval-2024 Task 7 subtask 3, our team Challenges aims on using contrastive loss to increase the understanding of the number from their different expression, and knows to identify between different number and its respective expression. This system description paper uses T5 and BART as the baseline models, comparing its result with and without the constrative loss. The result shows that BART with contrastive loss have surpassed all the models, and its performance on the number accuracy has the highest performance among all.
\end{abstract}

\section{Introduction}

This paper is a description of the methods we have applied for our implementation on this year's SemEval Task 7, NumEval: Numeral-Aware Language Understanding and Generation. SemEval is an annual workshop which is consisted of various natural language processing shared tasks. Teams that join the tasks is required to design systems that could enhance the understanding or improve results on various kinds of semantic evaluation challenge. The task we decide to join was task 7, NumEval: Numeral-Aware Language Understanding and Generation \citep{huang2023numhg}. Specifically, we focus on the second subtask from the third task in NumEval, which centers on generating proper news headline based on the provided news articles. Different from article summary, a headline must condense the essence from the full length article. Although the encoder-decoder language models nowadays has excelled on generation of the text based on the ROUGE metric, it still fails on providing precise numeral generation in headlines owing to the fact that the representation of the number may differ in various kinds of forms. Therefore, the goal of this task is to enhance the accuracy of the model in the generation of the number from the headline of a news article. 

While the numeral expression in the article consists of text and numbers, our system aims to use the technique of contrastive learning. With this technique, it is possible to help the model enclose the similarity between the number and its text expression, and enlarge the difference between different numbers (and its respective expression).

The evaluation of the performance is divided into two parts, one is to evaluate the accuracy of the predicted number, the other is to evaluate the accuracy of the word prediction, which uses ROUGE, BERTScore, and MoverScore to evaluate the results.

Two datasets are included in the task. One is the dry-run dataset that is provided on the official site. It contains a total of 100 instances; the other is the official training set that is provided after the registration of the task, which includes a total of 21157 instances. All of them have the same data structure. Each instance contains both the news article that includes the date of release, and its respective headline. Each team is expected to generate a precise news title from its respective news article.

Numbers are one of the most important element among medical, business, and legal article, and it could be dangerous if the large language model has misunderstood the content of the article. The finding of this paper could further discover a better way for large language model to detect and reason the numbers from the respective article, and thus generate the accurate headline with correct number.

Our system description is divided into three main section. First, we present the main approach that we used as the final submission result in detail, including the preparation of the dataset, its augmentation, and the structure of the model in contrastive learning that we designed in order to solve the task. Hence, we talk about the adjustment of the model, including parameter optimization on the model. Finally, we present our experimental result. In the end, we discuss about the possible future work, and conclude with a brief summary of our system.

\section{Related work}
Large Language Model (LLM) like ChatGPT has been long commented with its brittleness on the ability of numerical reasoning. When the questions presented in the varying textual form (comprising words and numbers), LLM would result in inconsistent performance \citep{ahn2024large}. In \citep{huang2023numhg}'s work, it shows that although large language model excelled based on ROUGE metrics, it still fails to generate precise numeral in headline.

Researchers have applied contrastive learning in natural language processing. To generate headline with different author style, \citep{liu2022contrastive} has applied contrastive learning to integrate the stylistic feature of the author into the model. This research hence inspires us to use contrastive learning on integrating the numeral features in different text form to the model, in order to let model identify the correct number from the news article.
\section{Methodology}
In this section, we propose the methodology of our system. It mainly consists of three parts. First, we will talk about the augmentation of the data. Secondly, the model training and fine-tuning, and finally, parameter optimization.
\subsection{Data pre-processing and augmentation}

A total of two vectors are used in our experiment. The first vector enhance the model understanding of different expression on number. One is to change all the text expression of number into numeral expression, and vice versa. While it encloses the similarity between the different expression of the same number, we call it positive distorted sample. With this change, model can better learn the different expression from the same number. We manually annotated the dataset to change the number into different kind of form. For example, if the number is \textit{1000}, then it would be transferred into \textit{1K}. The purpose of this is to increase the range of the understanding of the number in all forms.


\begin{table}
\fontsize{7.5}{11}\selectfont
\centering
\begin{tabular}{l}
\hline
\textbf{Positively Distorted} \\
\hline
\texttt{30K Walmart Part-Timers to Lose Health Insurance.} \\
\texttt{Thirty thousand Walmart part-timers to Lose Health Insurance} \\
\hline
\textbf{Negatively Distorted} \\
\hline
\texttt{Dax Shepard: Wedding to Kristen Bell Cost \$142.} \\
\texttt{Dax Shepard: Wedding to Kristen Bell Cost eight hundred } \\
\end{tabular}
\caption{Examples of positively and negatively distorted headlines}
\label{tab:positively_distorted}
\end{table}
The other vector, on the other hand, serves as the role that teaches the model to identify different numbers. It helps to enlarge the difference between the different numeric expressions, we call it negative distorted sample. Therefore, we also apply ChatGPT with the prompt\footnote{You are the examiner. Examine the text. If the number in any form appears in the text, change the number into another number. Return the revised text only.} to change the number from every news article.


\subsection{Encoder}
Transformer Seq2Seq Model \citep{vaswani2017attention} revolutionized the field of sequence-to-sequence learning. The implementation of the self-attention mechanism allowed weighting of the importance of different input tokens during the generation of each output token. The creation of multi-head attention enhanced the ability of the model to capture the diverse relationship between tokens. Based on transformer model advantages, the pre-trained BART-base model \citep{lewis-etal-2020-bart} was selected as an encoder for headline representation creation. For the comparison, the T5 \citep{raffel2020exploring}  model was also utilized.
\subsection{Models}
In this section, we aim to delineate the types of models we trained and the portion of data utilized for training. The first model, referred to as \textbf{BART(sub)}, represents the outcomes of a model submitted for evaluation. In this model, the hyperparameters of the pre-trained BART model with Contrastive Learning (CL) were fine-tuned. Following the submission, our focus shifted towards improving and adjusting the Contrastive Learning approach. The subsequent model, named \textbf{BART with CL}, was trained using improved contrastive learning techniques. However, due to resource constraints, it was trained solely on \textbf{1000} instances of data.


\subsection{Contrastive Loss}
Our proposed Contrastive Learning enhanced model was implemented on End-to-End Seq2Seq generation model. For the implementation of CL, positive and negative samples of headlines were explicitly constructed. Given news and headline, we trained End-to-End Seq2Seq models to generate headlines based on ground truth headlines. For the CL part, the model uses news, 2 positive and 2 negative headline samples. Given that our main task is specifically numerical aware headline generation, the sampling method was chosen to put more attention to numbers. That is why during the model training, the samples for the batch were exclusively formed from the 2 positive and 2 negative distortions of the same headline. This encourages the model to preserve the semantic content of the headline while allowing variations in numerical values. By explicitly focusing on numerical distortions in the loss function, the model learns to generate headlines that are robust to variations in numerical values. The loss function of the positive pair of examples (i, j) is defined as:
\\


\\
\begin{align}\label{eq:loss}
    {\cal L}_{s} = -\lambda\log\frac{\exp(\tau^{-1}sim({z}_i,{z}_j))}{\sum_{s} I(s \neq i)\exp(\tau^{-1}sim({z}_i,{z}_j))})
\end{align}
\\

where:
$I(\cdot)$\ is an indicator function such that $I(s \neq i)=1$ and $I(s = i)=0$, and $\tau$ is a temperature parameter and  $(s)$ is an index variable representing the current sample being considered during the training process.

This loss function penalizes the model if the distance between the news and positive headline embeddings is not closer than the distance between the news and negative headline embeddings by a certain margin.
The final loss function for the numeric headline generation task will consist of a combination of model loss  $L_{\text{model}}$ \ and contrastive loss $L_{\text{CL}}$ \ where the $\beta\ $ is hyperparameter.
Model Loss is the loss calculated from the model's forward pass using the ground truth labels. 
\[
\text{Loss} = L_{\text{model}} + \beta \times L_{\text{CL}}
\]



\subsection{Evaluation Metrics} 
For the evaluation of trained models, the automatic evaluation metric that the task organizer proposed was utilized. It consists of the ROUGE metric, incorporating ROUGE-1, ROUGE-2, and sentence-level ROUGE-L. For the BERTScore it incorporates BERT Precision, BERT Recall, and BERT F1. Also, the overall, copy and reasoning numerical accuracies were calculated.

\subsection{Implementation and Hyperparameters}
Due to resource constraints, all our models were trained only on \textbf{1000} instances of training data. For each model, we established the maximum length for both the article and the target headline, with values set at 512 and 16 respectively. In our trials, we adapt BART-large, T5-large and our newly introduced CL-augmented model. When tackling the headline generation task, we utilize beam search with a beam size of 8 and configure our batch size to 4. For the contrastive loss, the margin was set as 0.5, and $\beta\ $ of 0.5 was selected. We apply the Adam optimizer with a learning rate of $5 \times 10^{-6}$. The models undergo training for 10 epochs, with the validation set used to assess performance. All experiments were conducted on the Kaggle T4 GPU.

\section{Experimental Results}
Table 1 shows the performance from different headline generation models evaluated by ROUGE score. BART model trained with contrastive loss has achieved 35.74, which is higher than other baseline models, showing its effectiveness on headline generation. Comparing BART with and without contrastive loss (CL), we observe a notable improvement in ROUGE scores when contrastive loss is incorporated during training. BART with CL achieves the highest ROUGE-1 at 40.91 and ROUGE-2 at 17.49 scores. Results indicate that contrastive loss regularization enhances the model's ability to generate headlines with higher lexical overlap and coverage.

Table 2 is the BERTScore for each headline generation model. Its BERT F1 score of 41.77 reflects strong semantic similarity to reference summaries, indicating robust performance across both lexical and semantic dimensions. 

\begin{table}[h]
\centering
\begin{tabular}{l c c c}
\hline
\textbf{Model} & \textbf{Rouge-1} & \textbf{Rouge-2} & \textbf{Rouge-L} \\
\hline

\texttt{BART} & {38.48} & \ {15.18} & \ {33.35 } \\
\texttt{T5 with CL} & {36.72} & \ {14.31} & \ {32.58}\\
\texttt{BART(sub)}& {31.22} & \ {12.23} & \ {26.86}\\
\texttt{BART with CL} & \textbf{40.91} & \ \textbf{17.49} & \ \textbf{35.74}\\
%31.220	12.235	26.859
\hline

\end{tabular}
\caption{ROUGE scores of Headline
Generation Models}
\label{tab:example}
\end{table}

\begin{table}[h]
\centering
\begin{tabular}{l c c c}
\hline
%P: 0.356460 R: 0.458474 F1: 0.407268
\textbf{Model} & \textbf{P} & \textbf{R} & \textbf{F1} \\
\hline

\texttt{BART} & {33.4} & \ {45.61} & \ {33.46 } \\
%P: 0.354971 R: 0.399091 F1: 0.377181
\texttt{T5 with CL} & {35.50} & \ {39.90} & \ {37.72}\\
%19.530	47.559	33.132
\texttt{BART(sub)}& {19.53} & \ {	\textbf{47.56}} & \ {33.13}\\
\texttt{BART with CL} & {\textbf{36.91}} & \ {46.67} & \ \textbf{41.77 }\\
\hline
\end{tabular}
\caption{BERT scores of Headline
Generation Models}
\label{tab:example2}
\end{table}


\begin{table}[h]
\centering
\begin{tabular}{l c c c}
\hline

\textbf{Model} & \textbf{Overall} & \textbf{Copy} & \textbf{Reasoning} \\
\hline

\texttt{BART with CL} & {72.956} & \ {82.170} & \ {56.176 }\\
\hline
\end{tabular}
\caption{Numerical accuracy evaluation results }
\label{tab:example3}
\end{table}
Additionally, BART with CL exhibits a BERT Precision of 36.91 and a BERT Recall of 46.67, further emphasizing its balanced performance in capturing semantic content accurately.
\section{Conclusion}
Resolving Task 7 at SemEval-2024 as team Challenges, we applied contrastive learning techniques on several models, in order to see which obtain the model with highest performance. In the final submission we obtained the highest number accuracy in the COPY category, up to 82.170, and got the second place in overall score, also up to 72.956. 
In our human evaluation process, our headline generation model achieved the third-highest level of numerical accuracy by reaching 1.70 score. It thus proves that our approaches can help train the model in numerical reasoning and numerical headline generation. However, this model is merely trained on the \textbf{small part of dataset}. Therefore, in future work, it is suggested that more instances might help all the conventions. If we enlarge the data size, it is possible that the performance may get higher. Augmenting positive and negative headline samples artificially may enhance the effectiveness of numeric-based headline generation. In our future work, we are planning to generate positive and negative samples for the whole dataset, and train models. 









% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}

\end{document}
