\pdfoutput=1
\documentclass[11pt]{article}
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
\usepackage{booktabs}

\usepackage{subfig, graphicx} 

\usepackage{microtype}

\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{UMUTeam at SemEval-2024 Task 6: Leveraging zero-shot learning for detecting hallucinations and related observable overgeneration mistakes}



\author{Ronghao Pan \\
  Facultad de Informática, Universidad de Murcia, Campus de Espinardo, 30100 Murcia, Spain
  \texttt{ronghao.pan@um.es} \\\And
  Second Author \\
  Affiliation / Address line 1 \\
  Affiliation / Address line 2 \\
  Affiliation / Address line 3 \\
  \texttt{email@domain} \\}

\begin{document}
\maketitle
\begin{abstract}
In this manuscript, we describe the UMUTeam's participation in SemEval-2024 Task 6, a joint task aimed at detecting grammatically correct NLG output containing incorrect semantic information, with or without access to the model that produced the output. The task included three related subtasks with different model setups. Our approach is based on exploiting the zero-shot classification capability of state-of-the-art LLMs, such as LLaMa-2, Tulu and Mistral, through prompt engineering. Our system ranked eighteenth in the model-aware setup with an accuracy of 0.784000 and 29rd in the model-agnostic setup with an accuracy of 0.769333.
\end{abstract}

\section{Introduction}
Recently, the emergence of Large Language Models (LLMs) has brought about a paradigm shift in Natural Language Processing (NLP), leading to unprecedented advances in language understanding \cite{huang2023c} and reasoning \cite{zhang2023benchmarking}. Generally, LLMs refer to a set of general-purpose models based on the Transformer architecture and pre-trained on large text corpora, such as GPT-3 \cite{brown2020language}, LLaMa \cite{touvron2023llama}, PaLM \cite{chowdhery2023palm} and GPT-4 \cite{achiam2023gpt}. By scaling the amount of data and model capacity, LLMs demonstrate incredible emergent capabilities, typically including In-Context Learning (ICL) \cite{brown2020language}, chain-of-thought prompting \cite{wei2022chain}, and instruction following \cite{peng2023instruction}.

The modern panorama of Natural Language Generation (NLG) is plagued by two related problems. On the one hand, current models tend to produce inaccurate but fluent output, and on the other hand, metrics are better at describing fluency than correctness. This leads neural networks to "hallucinate", i.e. to produce fluent but incorrect output, and much research is currently being done to automatically detect this. For many NLG applications, however, the correctness of an output is critical to the task. For example, producing a translation that sounds plausible but is inconsistent with the source text jeopardises the usefulness of an automatic translation system.

Thus, this SHROOM task aims to detect grammatically correct outputs that contain incorrect semantic information (i.e. unsupported or inconsistent with the source input), with or without access to the model that produced the output. To this end, the organisers have adapted a post-hoc environment in which the models have already been trained and the outputs have already been produced, and the participants' task is a binary classification problem to identify cases of hallucinations, i.e. to detect grammatically correct outputs that contain incorrect or unsupported semantic information, in two different setups: model-aware and model-agnostic tracks.

For this task, we have employed the zero-shot learning approach using the latest LLMs such as LLaMa-2 \cite{touvron2023llama}, Mistral \cite{jiang2023mistral}, and Tulu \cite{ivison2023camels} to detect grammatically correct outputs containing incorrect semantic information through the prompt. The zero-shot technique refers to the ability of LLMs to perform tasks without being explicitly trained on them, implying that the model can generate responses or make predictions on topics or domains that have not been part of its explicit training. This is achieved by leveraging the general knowledge that the models have acquired during their massive pre-training on large text corpora.

Besides the system description, we make the following observations based on our experiments: 
\begin{enumerate}
    \item The latest LLMs are able to identify grammatically correct output containing incorrect semantic information in a zero-shot scenario.
    \item The zero-shot learning evaluation of the latest LLMs has shown that Tulu is the one best suited for this task.
\end{enumerate}

The remainder of this paper is organized as follows. Section \ref{sec:background} provides a summary of important details about the task setup.  Section \ref{sec:system-overview} offers an overview of our system for two subtasks. Section \ref{sec:experimental-setup} presents the specific details of our systems. Section \ref{sec:results} discusses the results of the experiments, and finally, the conclusions are shown in Section \ref{sec:conclusion} 

\section{Background}
\label{sec:background}
Natural Language Generation (NLG) is an area of AI and computational linguistics concerned with the automated generation of text in human language. The field covers a wide range of tasks, such as text generation for chatbots, automatic summarisation, machine translation, story generation, and others. NLG relies on models and algorithms that enable machines to understand and generate text that is coherent and intelligible to humans. However, current models tend to produce inaccurate but fluent output, while the metrics tend to describe fluency rather than correctness. This leads to models producing ``hallucinations'', i.e. generated content that appears nonsensical or unfaithful to the given source content. 

In general, hallucinations in natural language generation tasks can be divided into two main types \cite{ji2023survey}: intrinsic hallucinations and extrinsic hallucinations. Specifically, intrinsic hallucinations refer to the output of LLMs that conflict with the source content. Conversely, extrinsic hallucinations refer to LLM generations that cannot be verified from the source content. But, in the era of large language models, the versatile capabilities of these models have facilitated their widespread use in various domains, highlighting the limitations of the specific categorisation paradigms of the existing task.

For this reason, the organisers have proposed this task, which aims to automatically detect hallucinations and related observable overgeneration errors. To achieve this, they have provided a collection of checkpoints, inputs, references and outputs from systems covering three different NLG tasks: definition modelling (DM), machine translation (MT) and paraphrase generation (PG), trained with different levels of accuracy.The development set includes binary annotations from at least five different annotators and a majority vote gold label.


The generalisability of LLMs is very attractive for social scientists, as it allows us to use state-of-the-art methods and adapt them to our specific goals. For example, an LLM trained in multilingual texts could perform translations without being explicitly trained to do so (known as zero-shot capability). Analysts can further guide models by providing them with examples of the input and the desired output, known as few-shot learning. For example, we might show a sentence and its translation before asking it to translate another sentence. This additional information should help improve the quality of the output. For text classification tasks, the ability to make such predictions with little or no training makes these models especially promising for empirical research, as they have the potential to perform accurately without the need for costly and time-consuming annotation procedures. 

Therefore, we took advantage of this zero-shot classification capability of LLMs to detect hallucinations and related observable overgeneration errors. The following models were evaluated:
\begin{itemize}
    \item \textbf{Mistral} \cite{jiang2023mistral}. In the ever-evolving field of Natural Language Processing (NLP), the race for higher model performance often requires an escalation in model size. However, this scalability tends to increase computational cost and inference latency, raising the barriers to implementation in practical real-world scenarios. Mistral 7B demonstrates that a carefully designed language model can deliver high performance while maintaining efficient inference. Mistral 7B outperforms the 13 billion parameter LLaMa-2 model on all benchmarks. In addition, Mistral 7B approaches the coding performance of Code-Llama 7B without sacrificing performance on non-code benchmarks.
    \item \textbf{LLaMa-2} \cite{touvron2023llama}. It is a family of pre-trained and fine-tuned LLMs, Llama 2 and Llama 2-Chat, at scales of up to 70B parameters. In the set of utility and security benchmarks tested, Llama 2-Chat models generally outperform existing open-source models. For our system, we have used an instructively fine-tuned version of LLaMa-2 with 7B parameters from the Orca \cite{mukherjee2023orca} set called ``stabilityai/StableBeluga-7B\footnote{\url{https://huggingface.co/stabilityai/StableBeluga-7B}}''.
    \item \textbf{Tulu} \cite{ivison2023camels}. It's a family of pre-trained and fine-tuned LLMs. Unlike other existing LLMs, their newly distilled data mixtures have been shown to significantly improve downstream performance over instruction and preference datasets available just six months ago, with their new mixture outperforming their previous mixture by an average of 8\%. In addition, their Direct Preference Optimization (DPO) fine-tuned version scales to 70 billion parameter models and significantly improves open-response generation metrics without compromising model performance, improving AlpacaEval performance by an average of 13\% across all model scales. For this task, we have evaluated the DPO version of Tulu with 7 billion parameters called ``tulu-2-dpo-7b\footnote{\url{https://huggingface.co/allenai/tulu-2-dpo-7b}}''
\end{itemize}

\section{System overview}
\label{sec:system-overview}
Figure \ref{fig:arquitectura} shows the architecture of our system. We can see that we have introduced a specific prompt for each LLM to generate a response with the desired structure. Then we have a module called ``filter'' that extracts a binary response based on the response and the correlation value.

\begin{figure}[htb]
\centering
\includegraphics[width=0.5\textwidth]{arquitectura.png}
\caption{System architecture approach}
\label{fig:arquitectura}
\end{figure}

\subsection{Prompt}
\label{sec:prompt}
The prompt in the context of LLMs refers to a specific input provided to the model to elicit a desired response or to guide its text generation. This prompt can be a sentence, a question, or even a fragment of text that sets the context or direction for the model's text generation. For our system, we have used prompt engineering, which involves the design and careful wording of these prompts to elicit specific model responses and optimally influence the model's response. Figure \ref{fig:prompts} shows the prompts used for each LLM.

\begin{figure}[htb]
\centering
\includegraphics[width=0.5\textwidth]{prompts-examples.png}
\caption{Examples of prompts for each LLMs}
\label{fig:prompts}
\end{figure}

We can see that each LLM has its own control tokens to indicate which parts are system control sequences and which parts are user questions. For example, in LLaMa-2 ``\#\#\# System'' is used to indicate the control sequence, and ``\#\#\# User'' is used to indicate the user question. However, Mistral and Tulu do not have tokens to indicate system control sequences, but we can append the control sequence to the user question. In our system they all have the same prompt structure:
\begin{itemize}
    \item \textbf{System control sequence}. It is used to specify the context and instruct the model to respond only with ``yes'' or ``no'' at the beginning of the response. 
    \item \textbf{User question}. It is used to introduce the text and specify the question "the Sentence supported by the Context above".
\end{itemize}

Once the response generated by the LLMs is obtained, it is passed through the filter module, which identifies the first word of the response and classifies it as ``Hallucination'' or ``Not Hallucination''. To obtain the correlation value, we have used the same approach as the baseline provided by the organisers, which consists of extracting the log probability value of the first token of the response generated by the LLM.

\section{Experimental setup}
\label{sec:experimental-setup}
In this section we explain the dataset used, the hyperparameters used in the LLMs to generate responses, and details of the metrics used by the organisers for evaluation.

In this task, the organisers have provided us with 3 datasets: unlabelled train dataset, unlabelled model agnostic and model aware training data; trial labelled model agnostic and model aware training data, and validation data. In this case, we have only used the validation set to evaluate the performance of different models using a zero-shot classifier approach. In Figure \ref{fig:distribution} we can see the distribution of the validation set.

\begin{figure}[htb]
\centering
\includegraphics[width=0.5\textwidth]{Validation set distribution.pdf}
\caption{Validation set distribution}
\label{fig:distribution}
\end{figure}

The hyperparameters used in the LLMs for response generation are 0.95 for top\_p, 0 for top\_k, 256 for max\_new\_tokens and the default temperature for each LLM. And the evaluation metrics used are accuracy for binary classification and Rho to evaluate correlation.

\section{Results}
\label{sec:results}
Table \ref{tab:results-evaluation} shows the results of different LLMs in the validation set with two different configurations proposed by the organisers: model-aware and model-agnostic tracks. Therefore, the system created has to identify when an output produced by NLG is grammatically correct but contains incorrect or incompatible semantic information, inconsistent with the source input, either with or without access to the model that produced the output. From the table, we can see that the Tulu model with 7 billion parameters, trained with DPO, performed best in both the model-aware and model-agnostic configurations. It obtained an accuracy of 0.766467 and a Rho of 0.521104 in the model-aware configuration and an accuracy of 0.737475 and a Rho of 0.553962 in the model-agnostic metric. 
\begin{table}[htb!]
\centering
\caption{Results obtained with different LLMs in the validation set.}\label{tab:results-evaluation}
\begin{tabular}{lrr}
\toprule
\textbf{LLM} &  \textbf{Accuracy} &  \textbf{Rho}   \\
\midrule
\multicolumn{3}{c}{\textbf{Aware}} \\
\midrule
      Mistral &    0.522954 & 0.345239 \\
      Tulu &     \textbf{0.766467} & \textbf{0.521104}  \\
      LLaMa-2 &     0.668663 & 0.487483  \\
\midrule   
\multicolumn{3}{c}{\textbf{Agnostic}} \\
\midrule   
      Mistral &    0.503006 & 0.229504 \\
      Tulu &     \textbf{0.737475} & \textbf{0.553962} \\
      LLaMa-2 &     0.655310 & 0.521414  \\
\bottomrule
\end{tabular}
\end{table}

Therefore, we used the Tulu model to score the test set. In Table \ref{tab:ranking} we can see the official ranking for the task. We achieved the eighteenth best result out of a total of 46 teams in the model-aware setup, with a precision of 0.784000 and a Rho of 0.506895. Comparing our result to the first place result, we see that our model is 2.866\% worse in precision and 19.25\% worse in Rho. Regarding the model independent setup, our system obtained the nineteenth best result out of 49 participants. From Table \ref{tab:ranking} we can see that our system obtained a precision of 0.769333, which is 7.8\% worse than the first placed team (ahoblitz), and a Rho of 0.560945, which is 20.86\% worse than the first placed team.


\begin{table}[htb!]
\centering
\caption{Official raking table}\label{tab:ranking}
\resizebox{\columnwidth}{!}{
\begin{tabular}{lrrr}
\toprule
\textbf{LLM} &  \textbf{Rank} & \textbf{Accuracy} &  \textbf{Rho}   \\
\midrule
\multicolumn{4}{c}{\textbf{Aware}} \\
\midrule
      HaRMoNEE & 1 &    0.812666 & 0.699316 \\
      ahoblitz & 2 &     0.806000 & 0.714712  \\
      TU Wien & 3 &     0.806000 & 0.707192  \\
      ... & ... &     ... & ...  \\
      UMUTeam & 18 &     \textbf{0.784000} & \textbf{0.506895}  \\
\midrule   
\multicolumn{4}{c}{\textbf{Agnostic}} \\
\midrule   
      ahoblitz & 1 &   0.847333 & 0.769512 \\
      OPDAI &  2 &   0.836000 & 0.732195 \\
      HIT\_WL & 3 &   0.830666 & 0.76770 \\
      ... & ... &     ... & ...  \\
      UMUTeam & 29 &     \textbf{0.769333} & \textbf{0.560945}  \\
\bottomrule
\end{tabular}
}
\end{table}

\subsection{Error analysis}
\label{sec:error-analysis}
To perform an error analysis of our system, we extracted the confusion matrix from Tulu on the test set of the two configurations (model-aware and model-agnostic). A confusion matrix is a tool used in error analysis, especially in classification problems, that shows the performance of a model in predicting the true classes of a dataset versus the classes predicted by the model. The matrix shows the number of correct and incorrect predictions in each class, allowing the identification of the types of errors made by the model, such as false positives, false negatives, true positives and true negatives.

In Figure \ref{fig:aware-cm} we can see the confusion matrix of Tulu from the test set in the model-aware setup, and we can see that the model tends to confuse hallucinations with non-hallucinations with a probability of 41.38\%. However, it performs very well in detecting ``Not Hallucination'' with a probability of 89.88\%. Regarding the model-agnostic setup, our model tends to confuse hallucinations with "not hallucination" with a probability of 32.08\%, but is able to identify ``Not Hallucination'' with an accuracy of 83.13\%.

The TuLu model from the test set in the model-aware setup has obtained a total of 324 misclassifications, of which 165 are of the definition modelling type, 100 are of the machine translation type and 59 are of the paraphrase generation type. Therefore, we have a total of 165 misclassifications with Flan-T5\footnote{ltg/flan-t5-definition-en-base} model, 100 with the NLLB\footnote{facebook/nllb-200-distilled-600M model} and 59 with the Pegasus Paraphrase\footnote{tuner007/pegasus\_paraphrase} model. In order to know the most frequent topic that the model comments on the classification error, we used the BERTopic model to identify and group topics in the context of the failed cases. In Figure  we can see the 7 topics in which the TuLu model usually misidentifies.

\begin{figure}[htb]
\centering
\includegraphics[width=0.5\textwidth]{aware-cm.png}
\caption{Confusion matrix of Tulu with test dataset in model-aware setup.}
\label{fig:aware-cm}
\end{figure}

\begin{figure}[htb]
\centering
\includegraphics[width=0.5\textwidth]{agnostic-cm.png}
\caption{Confusion matrix of Tulu with test dataset in model-agnostic setup.}
\label{fig:agnostic-cm}
\end{figure}

\section{Conclusion}
\label{sec:conclusion}

In this paper, we describe the UMUTeam's participation in the common task 6 of SemEval 2024. The objective was to develop models for detecting grammatically sound output by NLG that contains incorrect semantic information (i.e. unsupported or inconsistent with the source input), with or without having access to the model that produced the output. For this task, we have employed the zero-shot learning approach using the latest LLMs such as LLaMa-2 \cite{touvron2023llama}, Mistral \cite{jiang2023mistral}, and Tulu \cite{ivison2023camels} to detect grammatically correct outputs containing incorrect semantic information through the prompt. The 7-billion parameter Tulu model trained with DPO performed best in the evaluation set. Using this model, we ranked eighteenth in the model-aware setup with an accuracy of 0.784000 and nineteenth in the model-agnostic setup with an accuracy of 0.769333.



\section*{Acknowledgements}

% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}

\appendix


\end{document}
