\pdfoutput=1
\documentclass[11pt]{article}
\usepackage[preprint]{acl}
\usepackage{times}
\usepackage{latexsym}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{booktabs}
\usepackage{subfig, graphicx} 
\usepackage{microtype}
\usepackage{inconsolata}

\title{UMUTeam at SemEval-2024 Task 6: Leveraging Zero-Shot Learning for Detecting Hallucinations and Related Observable Overgeneration Mistakes}

\author{
  Ronghao Pan\(^1\),
  José Antonio García-Díaz\(^1\),
  Tomás Bernal-Beltrán\(^1\),\\
  \textbf{Rafael Valencia-García}\(^1\)\\
\(^1\) Facultad de Informática, Universidad de Murcia, Campus de Espinardo, 30100 Murcia, Spain\\
{\{\tt ronghao.pan, joseantonio.garcia8, tomas.bernalb, valencia\}@um.es}  \\ 
}

\begin{document}
\maketitle
\begin{abstract}
In these working notes we describe the UMUTeam's participation in SemEval-2024 shared task 6, which aims at detecting grammatically correct output of Natural Language Generation with incorrect semantic information in two different setups: model-aware and model-agnostic tracks. The task is consists of three subtasks with different model setups. Our approach is based on exploiting the zero-shot classification capability of the Large Language Models LLaMa-2, Tulu and Mistral, through prompt engineering. Our system ranked eighteenth in the model-aware setup with an accuracy of 78.4\% and 29th in the model-agnostic setup with an accuracy of 76.9333\%.
\end{abstract}

\section{Introduction}
Recently, the emergence of Large Language Models (LLMs) has brought about a paradigm shift in Natural Language Processing (NLP), leading to unprecedented advances in Natural Language Understanding (NLU) \cite{huang2023c} and Reasoning \cite{zhang2023benchmarking}. In general, LLMs refer to a set of general-purpose models based on the Transformer architecture and pre-trained on large text corpora, such as GPT-3 \cite{brown2020language}, LLaMa \cite{touvron2023llama}, PaLM \cite{chowdhery2023palm} and GPT-4 \cite{achiam2023gpt}. By scaling the amount of data and model capacity, LLMs demonstrate incredible emergent capabilities, typically including In-Context Learning (ICL) \cite{brown2020language}, chain-of-thought prompting \cite{wei2022chain}, and instruction following \cite{peng2023instruction}.

Natural Language Generation (NLG) faces two related challenges. First, current models often produce output that is fluent but inaccurate. Second, the metrics used to evaluate the LLMs performance prioritize fluency over correctness, exacerbating the problem of ``hallucination'', in which LLMs produce fluent but incorrect output. Consequently, significant research in underway to automatically detect such errors. In many NLG applications, output correctness is paramount, as in cases such as machine translation, where producing a plausible but inconsistent translation compromises the utility of the system. 

Thus, the SHROOM shared-task focuses on identifying grammatically correct outputs that contain incorrect semantic information, regardless of whether the model producing the output is accessible or not \cite{SHROOM}. To this end, the organizers have adapted a post-hoc environment in which the models have already been trained, and the outputs have already been produced. The participants' task is a binary classification problem to identify cases of hallucinations, i.e. to detect grammatically correct outputs that contain incorrect or unsupported semantic content, in two different setups: model-aware and model-agnostic tracks.

To address the SHROOM challenge, our team used a zero-shot learning (ZSL) approach with LLaMa-2 \cite{touvron2023llama}, Mistral \cite{jiang2023mistral}, and Tulu \cite{ivison2023camels} LLMs to detect grammatically correct output that contains incorrect semantic information through the prompt. The ZSL technique refers to the ability of LLMs to perform tasks without being explicitly trained on them, meaning that the model can generate responses or make predictions on topics or domains that were not part of its explicit training. This is achieved by exploiting the general knowledge that the models have acquired during their massive pre-training on large text corpora.

During our experiments, we observed that these LLMs were able to identify hallucinations. In particular, Tulu is the one best suited for this task.

The rest of this paper is organized as follows. Section \ref{sec:background} provides a summary of important details about the task setup. Section \ref{sec:system-overview} gives an overview of our system for two subtasks. Section \ref{sec:experimental-setup} presents the specific details of our systems. Section \ref{sec:results} discusses the results of the experiments, and finally, the conclusions are presented in Section \ref{sec:conclusion}.


\section{Background}
\label{sec:background}
NLG is a branch of Artificial Intelligence (AI) and computational linguistics that deals with the automated generation of text in human language. NLG covers a wide range of tasks, such as text generation for chatbots, automatic summarization, machine translation, story generation, and others. NLG relies on models and algorithms that enable machines to understand and generate text that is coherent and intelligible to humans. However, current models can produce inaccurate but fluent output, while the metrics tend to describe fluency rather than correctness. This leads to models producing ``hallucinations'', i.e. generated content that appears nonsensical or unfaithful to the given source content. 

In general, hallucinations in NLG tasks can be divided into two main types \cite{ji2023survey}: intrinsic hallucinations and extrinsic hallucinations. On the one hand, intrinsic hallucinations refer to the output of LLMs that conflict with the source content. On the other hand, extrinsic hallucinations refer to LLM generations that cannot be verified from the source content. 

The SHROOM task aims to automatically detect hallucinations and related observable overgeneration errors. To achieve this, the organizers have provided a collection of checkpoints, inputs, references and outputs from systems covering three different NLG tasks: (1) definition modeling (DM), (2) machine translation (MT), and (3) paraphrase generation (PG), trained with different levels of accuracy. The development set includes binary annotations from at least five different annotators and a majority vote gold label.

The generalizability of LLMs is very attractive because it allows us to adapt state-of-the-art methods to specific goals. For example, an LLM trained on multilingual texts can perform translations without being explicitly trained to do so (known as zero-shot capability, ZSL). Another possibility is to guide models by providing them with examples of the input and the desired output (known as few-shot learning, FSL). For example, in \cite{garcia2023leveraging}, LLMs have shown good performance in a ZSL scenario for identifying hate speech in Spanish and English. In this sense, it is possible to ask for a sentence and its translation before asking it to translate another sentence. This additional information helps to improve the quality of the output. For text classification tasks, the ability to make such predictions with little or no training makes these models particularly promising for empirical research, as they have the potential to perform accurately without the need for costly and time-consuming annotation procedures. 

Therefore, we took advantage of this ZSL classification capability of LLMs to detect hallucinations and related observable overgeneration errors. 

\label{sec:system-overview}
\begin{figure*}[!htb]
\centering
\includegraphics[width=.75\textwidth]{arquitectura.png}
\caption{System architecture approach}
\label{fig:arquitectura}
\end{figure*}


The following models are evaluated:

\begin{itemize}
    \item \textbf{Mistral} \cite{jiang2023mistral}. Higher model performance often requires an escalation in model size. However, this scalability tends to increase computational cost and inference latency, raising the barriers to implementation in practical real-world scenarios. Mistral 7B is a high-performance LLM that maintains efficient inference. Mistral 7B outperforms the 13 billion parameter LLaMa-2 model on all benchmarks. In addition, Mistral 7B approaches the coding performance of Code-Llama 7B without sacrificing performance on non-code benchmarks.
    
    \item \textbf{LLaMa-2} \cite{touvron2023llama}. Llama 2 and Llama 2-Chat are pre-trained and fine-tuned LLMs, both at scales of up to 70B parameters. In several benchmarks tested, Llama 2-Chat models generally outperformed existing open-source models. For our system, we used an instructively fine-tuned version of LLaMa-2 with 7B parameters from the Orca \cite{mukherjee2023orca} set called ``stabilityai/StableBeluga-7B\footnote{\url{https://huggingface.co/stabilityai/StableBeluga-7B}}''.
    
    \item \textbf{Tulu} \cite{ivison2023camels}. TuLu is a family of pre-trained and fine-tuned LLMs. Unlike other existing LLMs, distilled data mixtures from TuLu have been shown to significantly improve downstream performance over instruction and datasets available, with a new mixture outperforming its predecessor by an average of 8\%. In addition, TuLu models use a fine-tuned version of Direct Preference Optimization (DPO) that scales to 70 billion parameter models and significantly improves open-response generation metrics without compromising model performance, improving AlpacaEval performance by an average of 13\% across all model scales. For this task, we have evaluated the 7 billion parameter DPO version of Tulu with called ``tulu-2-dpo-7b''.\footnote{\url{https://huggingface.co/allenai/tulu-2-dpo-7b}}
\end{itemize}


\section{System overview}
Figure \ref{fig:arquitectura} shows the architecture of our system. We can see that we have introduced a specific prompt for each LLM to generate a response with the desired structure. Then we have a module called ``filter'' that extracts a binary response based on the response and the correlation value.

\subsection{Prompt}
\label{sec:prompt}
The prompt in the context of LLMs refers to a specific input provided to the model to elicit a desired response or to guide the text generation. This prompt can be a sentence, a question, or even a fragment of text that sets the context or direction for the model's text generation. In our proposal, we use prompt engineering, which involves the design and careful wording of these prompts to elicit specific model responses and optimally influence the model's response. 

Figure \ref{fig:prompts} shows the prompts used for each LLM, in which we can see that each LLM has its own control tokens to indicate which parts are system control sequences and which parts are user questions. For example, in LLaMa-2 ``\texttt{\#\#\# System}'' is used to indicate the control sequence, and ``\texttt{\#\#\# User}'' is used to indicate the user question. However, Mistral and Tulu do not have tokens to indicate system control sequences, but we can append the control sequence to the user question. 

\begin{figure}[!htb]
\centering
\includegraphics[width=.47\textwidth]{prompts-examples.png}
\caption{Examples of prompts for each LLMs}
\label{fig:prompts}
\end{figure}

In our system, we have used the same prompt structure for all LLMs: (1) \textbf{System control sequence}. It is used to specify the context and instruct the model to respond only with ``yes'' or ``no'' at the beginning of the response; and (2) \textbf{User question}. It is used to introduce the text and specify the question ``the Sentence supported by the Context above''. Once the response generated by the LLMs is obtained, it is passed through the filter module, which identifies the first word of the response and classifies it as ``Hallucination'' or ``Not Hallucination''. To obtain the correlation value, we have used the same approach as the baseline provided by the organizers, which consists of extracting the log probability value of the first token of the response generated by the LLM.

\section{Experimental setup}
\label{sec:experimental-setup}
In this section, we explain the dataset used, the hyperparameters used in the LLMs to generate responses, and details of the metrics used by the organizers for evaluation.

In this task, the organizers provided participants with unlabeled train data, trial data, and validation data for both the model-aware and model-agnostic setups. We have only used the validation data to evaluate the performance of different models using a ZSL approach. Figure \ref{fig:distribution} displays the distribution of the validation set.

\begin{figure}[!tb]
\centering
\includegraphics[width=.5\textwidth]{Validation set distribution.pdf}
\caption{Validation set distribution}
\label{fig:distribution}
\end{figure}

The hyperparameters used in the LLMs to generate the response are 0.95 for top\_p, 0 for top\_k, 256 for max\_new\_tokens and the default temperature for each LLM. 

The evaluation metrics used are accuracy for binary classification and rho to evaluate correlation. The rho metric, commonly known as the rho correlation coefficient ($\rho$), is a statistical measure that evaluates the relationship between two ordinal variables. It is particularly useful when the variables are not continuous but are divided into ordered categories.

\section{Results}
\label{sec:results}
Table \ref{tab:results-evaluation} shows the results of different LLMs in the validation set with two different configurations: (1) model-aware and (2) model-agnostic tracks. Thus, the system has to identify when a text is grammatically correct but contains incorrect information inconsistent with the source input, either with or without access to the model that produced the text. We can see that the Tulu performed best in both the model-aware and model-agnostic configurations. It obtained an accuracy of 76.6467\% and a rho of 0.521104 in the model-aware configuration and an accuracy of 73.7475\% and a rho of 0.553962 in the model-agnostic metric. 

\begin{table}[tb!]
\centering
\caption{Results obtained with different LLMs in the validation set.}\label{tab:results-evaluation}
\begin{tabular}{lrr}
\toprule
\textbf{LLM} &  \textbf{Accuracy} &  \textbf{Rho}   \\
\midrule
\multicolumn{3}{c}{\textbf{Aware}} \\
\midrule
Mistral &    52.2954 & 0.345239 \\
Tulu    & \textbf{76.6467} & \textbf{0.521104}  \\
LLaMa-2 &     66.8663 & 0.487483  \\
\midrule   
\multicolumn{3}{c}{\textbf{Agnostic}} \\
\midrule   
Mistral &    50.3006 & 0.229504 \\
Tulu    & \textbf{73.7475} & \textbf{0.553962} \\
LLaMa-2 &    65.5310 & 0.521414  \\
\bottomrule
\end{tabular}
\end{table}

According to the results with development, we used Tulu in the task. Table \ref{tab:ranking} shows the official ranking for the task. We achieved the eighteenth best result out of a total of 46 teams in the model-aware setup, with a precision of 78.4\% and a rho of 0.506895. Compared to result to the best result, our model is 2.866\% worse in precision and 19.25\% worse in rho. Regarding the model independent setup, our system achieved the nineteenth best result out of 49 participants, with a precision of 76.9333\%, which is 7.8\% worse than the best team (ahoblitz), and a rho of 0.560945, which is 20.86\% worse than the best team.


\begin{table}[htb!]
\centering
\caption{Official raking table}\label{tab:ranking}
\begin{tabular}{lrrr}
\toprule
\textbf{LLM} &  \textbf{Rank} & \textbf{Accuracy} &  \textbf{Rho}   \\
\midrule
\multicolumn{4}{c}{\textbf{Aware}} \\
\midrule
HaRMoNEE & 1 &     81.2666 & 0.699316 \\
ahoblitz & 2 &     80.6000 & 0.714712  \\
TU Wien  & 3 &     80.6000 & 0.707192  \\
\multicolumn{4}{c}{\ldots} \\
UMUTeam & 18 &     \textbf{78.4000} & \textbf{0.506895}  \\
\midrule   
\multicolumn{4}{c}{\textbf{Agnostic}} \\
\midrule   
ahoblitz & 1 &   84.7333 & 0.769512 \\
OPDAI &  2 &     83.6000 & 0.732195 \\
HIT\_WL & 3 &    83.0666 & 0.767700 \\
\multicolumn{4}{c}{\ldots} \\
UMUTeam & 29 &     \textbf{76.9333} & \textbf{0.560945}  \\
\bottomrule
\end{tabular}
\end{table}

\begin{figure*}[htb]
\begin{minipage}[b]{.45\textwidth}
\centering
\includegraphics[width=.9\textwidth]{aware-cm.png}
\caption{Confusion matrix of Tulu with test dataset in model-aware setup.}
\label{fig:aware-cm}
\end{minipage}
\begin{minipage}[b]{.45\textwidth}
\centering
\includegraphics[width=.9\textwidth]{agnostic-cm.png}
\caption{Confusion matrix of Tulu with test dataset in model-agnostic setup.}
\label{fig:agnostic-cm}
\end{minipage}
\end{figure*}


\subsection{Error analysis}
\label{sec:error-analysis}
We perform an error analysis of our system. For this, we extracted the confusion matrix from Tulu on the test set of the two configurations (model-aware and model-agnostic). 

\begin{figure*}[!htb]
\centering
\includegraphics[width=\textwidth]{topic-aware.png}
\caption{The most frequent topics associated with misclassification in model-aware setup.}
\label{fig:aware-topic}
\end{figure*}

Figure \ref{fig:aware-cm} shows the confusion matrix of TuLu from the test set in the model-aware setup. This approach tends to confuse hallucinations with non-hallucinations with a probability of 41.38\%. However, it performs very well at detecting ``not hallucination'' with a probability of 89.88\%. Regarding the model-agnostic setup, our model tends to confuse hallucinations with "not hallucination" with a probability of 32.08\%, but is able to identify ``not hallucination'' with an accuracy of 83.13\%.

The Tulu model from the test set in the model-aware setup has obtained a total of 324 misclassifications, of which 165 are of the definition modeling type, 100 are of the machine translation type, and 59 are of the paraphrase generation type. Therefore, we have a total of 165 misclassifications with the Flan-T5\footnote{ltg/flan-t5-definition-en-base} model, 100 with the NLLB\footnote{facebook/nllb-200-distilled-600M} model, and 59 with the Pegasus Paraphrase\footnote{tuner007/pegasus\_paraphrase} model. In order to know the most common topic that the model comments on the classification error, we used the BERTopic model to identify and group topics in the context of the failed cases. In Figure \ref{fig:aware-topic} we can see the 7 topics in which the TuLu model usually misidentifies.

Regarding the model-agnostic setup, our approach has obtained a total of 346 misclassifications, of which 138 are of the definition modeling type, 107 are of the machine translation type, and 101 are of the paraphrase generation type. In contrast to the model-aware setup, there is an increase in the accuracy of the identification of definition modeling misclassifications, but a decrease in the identification of paraphrase generation misclassifications. Figure \ref{fig:agnostic-topic} shows the three most common topics associated with the classification errors. 



\begin{figure}[htb]
\centering
\includegraphics[width=0.7\textwidth]{topic-agnostic.png}
\caption{The most frequent topics associated with misclassification in model-agnostic setup.}
\label{fig:agnostic-topic}
\end{figure}


\section{Conclusion}
\label{sec:conclusion}
Here we describe the UMUTeam's participation in SHROOM (SemEval 2024), concerning the development of models for detecting grammatically correct output from NLGs, but with incorrect semantic information in two different setups: model-aware and model-agnostic tracks. We have used the ZSL approach with LLaMa-2 \cite{touvron2023llama}, Mistral \cite{jiang2023mistral}, and Tulu \cite{ivison2023camels} LLMs to detect output that contains incorrect semantic information through the prompt. Tulu performed best in the evaluation set. Using this model, we ranked eighteenth in the model-aware setup with an accuracy of 78.4\% and nineteenth in the model-agnostic setup with an accuracy of 76.9333\%.

As further work, we propose to investigate hallucination detection in the political domain. In politics, automated content generation can help politicians to generate text on a variety of political topics, which can help political campaigns, think tanks, and government agencies quickly produce tailored content. Hallucination detection can help to mitigate misleading or fabricated content. In this sense, we propose to generate political discourse that imitates politicians from different political wings \cite{politices2022} and to identify the generated hallucinations by different LLMs.


\section*{Acknowledgements}
This work is part of the research projects LaTe4PoliticES (PID2022-138099OB-I00) funded by MICIU/AEI/10.13039/501100011033 and the European Regional Development Fund (ERDF)-a way to make Europe and LT-SWM (TED2021-131167B-I00) funded by MICIU/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR. In addition, this work was funded by the Spanish Government, the Spanish Ministry of Economy and Digital Transformation through the Digital Transformation through the "Recovery, Transformation and Resilience Plan" and also funded by the European Union NextGenerationEU/PRTR through the research project 2021/C005/0015007. Mr. Ronghao Pan is supported by the "Programa Investigo" grant, funded by the Region of Murcia, the Spanish Ministry of Labour and Social Economy and the European Union - NextGenerationEU under the "Plan de Recuperación, Transformación y Resiliencia (PRTR)".


\bibliography{custom}

\appendix


\end{document}
