% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{array}
\usepackage{multirow}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[OT2,T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
\newcommand\textcyr[1]{{\fontencoding{OT2}\selectfont #1}}

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm (default) or larger.

% To produce a PDF file, pdf\LaTeX{} is strongly recommended (over original \LaTeX{} plus dvips+ps2pdf or dvipdf). Xe\LaTeX{} also produces PDF files, and is especially suitable for text in non-Latin scripts.

\title{DeepPavlov at SemEval-2024 Task 6: Detection of Hallucinations and Overgeneration Mistakes with an Ensemble of Transformer-based Models}

\author{Ivan Maksimov \and Vasily Konovalov \and Andrei Glinskii \\
  Moscow Institute of Physics and Technology\\
  \texttt{\{maksimov.ivan.v, vasily.konovalov, glinsky\}@phystech.edu}\\}

\begin{document}
\maketitle
\begin{abstract}

The inclination of large language models (LLMs) to produce mistaken assertions, known as hallucinations, can be problematic. These hallucinations could potentially be harmful since sporadic factual inaccuracies within the generated text might be concealed by the overall coherence of the content, making it immensely challenging for users to identify them. The goal of the SHROOM shared-task is to detect grammatically sound outputs that contain incorrect or unsupported semantic information. Although there are a lot of existing hallucination detectors in generated AI content, we found out that pretrained Natural Language Inference (NLI) models yet exhibit success in detecting hallucinations. Moreover their ensemble outperforms more complicated models.
\end{abstract}

\section{Introduction}

Over the past few years, Natural Language Generation (NLG) models have experienced substantial advancements, particularly due to transformer-based architectures like a Generative Pretrained Transformer (GPT)~\cite{radford2019language}. However, two interconnected issues challenge the field: firstly, the tendency of present neural systems to generate incorrect yet smooth outputs and, secondly, the inadequacy of existing metrics in evaluating accuracy over fluency. This causes NLG models to ``hallucinate'', i.e., produce fluent but incorrect outputs that we currently struggle to detect automatically~\cite{Ji_2023}.

The Shared-task on Hallucinations and Related Observable Overgeneration Mistakes (SHROOM) has been suggested to address this challenge. In particular, the SHROOM task aims at addressing the existing gap in assessing the semantic correctness and meaningfulness of NLG models.\footnote{\url{https://helsinki-nlp.github.io/shroom/}} Within the Shared task~\citep{mickus-etal-2024-semeval}, one needs to detect grammatically sound English output that contains incorrect semantic information (i.e.,unsupported or inconsistent with the source input) in case there is no labeled training data available.

We propose to address the SHROOM task by leveraging an ensemble of pretrained transformer-based Natural Language Inference (NLI) models. The NLI models are used to derive features of hallucination probabilities, and then a tree-based gradient boosting model~\cite{prokhorenkova2019catboost} provides a final decision. Our results indicate that NLI-based models can be effectively used to detect hallucinations. Moreover, the ensemble model highly outperforms the base estimators in correlation with annotators' decisions. 

To summarize, this work includes the following contributions:

\begin{itemize}
\item We conducted a systematic study, re-evaluating existing NLI models for hallucination detection tasks.

\item We trained an ensemble of NLI models to detect hallucination that correlates with human judgment.  
\end{itemize}

Additionally, we made the code publicly available.\footnote{\url{https://github.com/ivan-kud/semeval-2024-shroom}}

\section{Background}

Nowadays, it is well known that NLG models often generate coherent outputs that are not faithful to the given input, commonly referred as hallucinations~\cite{maynez2020faithfulness}. Hallucination has been studied in a wide range of tasks, including but not limited to summarization~\cite{huang2021factual}, dialogue generation~\cite{shuster2021retrieval} and a variety of other NLG tasks.

There are several benchmarks for hallucination detection. HaluEval includes 5,000 general user queries with ChatGPT responses and 30,000 task-specific examples from three tasks, i.e., question answering, knowledge-grounded dialogue, and text summarization~\cite{HaluEval}. FaithDial is a benchmark for hallucination-free dialogues by modifying hallucinated responses in the Wizard of Wikipedia (WoW) benchmarks~\cite{dziri2022faithdial}.

The SHROOM shared task organizers went one step further. The shared task was conducted with a newly constructed dataset of 4,000 model outputs labeled by five annotators each, including three NLP tasks: machine translation (MT), paraphrase generation (PG), and definition modeling (DM). Participants were asked to detect hallucinations in two different settings: a model-aware track where the organizers also provided a checkpoint to a model that generated the output and a model-agnostic track where they did not. The checkpoints are publicly available on HuggingFace.

All three NLG tasks are in English, with the exception of the input for the MT task, which is in Russian for the model-agnostic task and in many other languages for the model-aware task~\cite{mickus-etal-2024-semeval}.

\section{Dataset}

The dataset for the SHROOM challenge comprises a compilation of model-generated text entries with the aim to classify each output as either a hallucination of the generative model or not.

Information for the data sample includes the following fields: (i) \textit{src} -- the input text given to the generative language model; (ii) \textit{hyp} -- the generated textual output of the model; and (iii) \textit{tgt} -- the intended reference or the ground truth text that the model is supposed to generate; (iv) \textit{task} -- the task being solved; (v) \textit{labels} -- five labels, either \textit{"Hallucination"} or \textit{"Not Hallucination"} labeled by five annotators, and finally, (vi) \textit{p(Hallucination)} indicates the proportion of annotators that labeled the data sample as a hallucination.

The dataset was split in the following way: training data of 30,000 samples without annotations with 10,000 samples for each task; validation data of 499 labeled samples with 187, 187, and 125 samples for DM, MT, and PG tasks, respectively; and test data of 1,500 examples without annotations to evaluate and rank the results of the competitors with 563, 562, and 375 examples for DM, MT, and PG tasks, respectively.
Validation data sample is presented in Table~\ref{data-examples}.

\begin{table*}
\centering
\begin{tabular}{lp{50mm}p{30mm}p{30mm}r}
\hline
\textbf{Task} & \textbf{Input} & \textbf{Output} & \textbf{Reference} & \textbf{p(Halluc.)} \\
\hline
DM & Because redpillers are usually normies or <define> Chadlites </define> while incels are ugly low - value males that women feel innately repulsed by . & (slang) An incel. & (incel, \_, slang) A man of a slightly lower ranking on a scale of physical attractiveness than Chad. & 0.2 \\
\hline
MT & \textcyr{Ядрён батон!} & Nuclear Baton! & Blimey! & 1.0 \\
\hline
PG & Are we resolved? & So, what, we're in the clear now? & Do we have a deal? & 0.4 \\
\hline
\end{tabular}
\caption{Data examples from the validation sample of the model-agnostic track. Hallucination probability of 0.2 means that one of the annotators classified the example as a hallucination and the remaining four annotators classified it as not a hallucination. The resulting class label is determined by majority voting. For the DM task, the model had to define a word between two special tokens: <define> and </define>.}
\label{data-examples}
\end{table*}

All participants' submissions were evaluated using two criteria:
\begin{itemize}
\item Accuracy that the system reached on the binary classification.
\item Spearman correlation of the system's output probabilities with the proportion of the annotators labeling the item as a hallucination.
\end{itemize}

\section{Methodology}

NLI task determines whether a hypothesis follows a premise and classifies it as either entailment, contradiction, or neutral. Previous research showed that NLI can be successfully used for hallucination detection in summarization~\cite{gekhman2023trueteacher}. 

Our system is an ensemble of several NLI pre-trained transformer-based models. Each model defines a new feature set, then a tree-based gradient boosting final estimator is trained on top of these features. 

It's well known that in-domain training data are crucial for classifier
performance~\cite{konovalov_collecting_2016}. However, organizers do not provide us with the labeled training set, so we train final estimator on the labeled validation set.

The following is a list of NLI transformer-based models used in our pipeline:

\begin{itemize}

\item RoBERTa by~\citet{liu2019roberta} is a robustly optimized BERT that outperformed BERT on the MNLI task.
% \footnote{\url{hf.co/FacebookAI/roberta-large-mnli}}

\item Sentence-RoBERTa by \citet{reimers-2019-sentence-bert} that returns sentence embeddings. Thus, we calculate the probability of hallucination as \(1 - S\), where \(S\) is cosine similarity between the premise and hypothesis embeddings.
% .\footnote{\url{hf.co/sentence-transformers/nli-roberta-large}} 

\item DeBERTa by~\citet{manakul2023selfcheckgpt} uses an disentangled attention mechanism
and an enhanced mask decoder being pre-trained on MNLI-m/mm outperformed BERT and RoBERTa of comparable sizes.
% \footnote{\url{hf.co/potsawee/deberta-v3-large-mnli}}

\item T5 by~\citet{honovich-etal-2022-true-evaluating} was trained on a binary problem (entailment or contradiction) on the bunch of NLU datasets: SNLI~\cite{bowman2015large}, MNLI~\cite{williams2017broad}, Fever~\cite{thorne2018fever},
Scitail~\cite{khot2018scitail},
PAWS~\cite{zhang2019paws},
VitaminC~\cite{schuster2021get}.
% \footnote{\url{hf.co/google/t5_xxl_true_nli_mixture}}

\end{itemize}

In addition to the output of the before-mentioned models, we add as features the lengths of input, output and reference texts. Then we train CatBoost~\citep{prokhorenkova2019catboost} models as metamodels on top of these features. Besides CatBoost model, we also train  Random Forest~\citep{random-forest} implemented in scikit-learn library~\citep{scikit-learn} and LightGBM~\citep{NIPS2017_6449f44a}. CatBoost yields the best results among them.

\section{Experimental setup}

We do not use any preprocessing of input texts (premises and hypotheses). Neither do we use an unlabeled training set. So, the transformer-based models serve to obtain features from the validation and test sets, then the CatBoost metamodel is trained on the validation set and predicts the test set.

As for the CatBoost metamodel, we performed the following steps:

\begin{itemize}
\item We found the hyperparameters on the validation set by using Optuna~\citep{optuna_2019}. Stratified k-fold cross-validation~\footnote{\url{https://en.wikipedia.org/wiki/Cross-validation_(statistics)}} with 10 splits was used for the classification model and k-fold cross-validation with 10 splits -- for the regression model. The best parameters for the classification model for the model-agnostic task: \texttt{iterations = 216}, \texttt{learning\_rate = 0.010}, \texttt{depth = 12}, and for the model-aware task: \texttt{iterations = 129}, \texttt{learning\_rate = 0.005}, \texttt{depth = 9}. The best parameters for the regression model for the model-agnostic task: \texttt{iterations = 356}, \texttt{learning\_rate = 0.029}, \texttt{depth = 5}, and for the model-aware task: \texttt{iterations = 317}, \texttt{learning\_rate = 0.012}, \texttt{depth = 9}.
\item We evaluated the metrics on the validation sample using repeated stratified k-fold cross-validation with 10 splits and 5 repeats.
\item We trained it on the whole labeled validation set.
\item We predicted test set labels.
\end{itemize}

\section{Results}

The results on the test set for both model-agnostic and model-aware tracks are presented in Table~\ref{results}. There are scores for the baseline provided by organizers, best scores from the leader-board, individual transformer-based models and our system as a whole.

\begin{table*}[!htb]
\centering
\begin{tabular}{lrrrr}
\hline
\multirow{2}{*}{Model} & \multicolumn{2}{c}{model-agnostic} & \multicolumn{2}{c}{model-aware}\\
& \textbf{Accuracy} & \textbf{Corr.} & \textbf{Accuracy} & \textbf{Corr.}\\
\hline
\texttt{nli-roberta-large}&62.8&0.608&66.2&0.566 \\
\texttt{roberta-large-mnli}&73.7&0.611&73.0&0.549 \\
\texttt{deberta-base-mnli}&72.8&0.617&73.1&0.597\\
\texttt{deberta-large-mnli}&75.7&0.701&75.5&0.688\\
\texttt{deberta-xlarge-mnli}&73.5&0.699&74.4&0.681\\
\texttt{deberta-v2-xlarge-mnli}&74.4&0.711&74.7&0.677\\
\texttt{deberta-v2-xxlarge-mnli}&76.1&0.729&75.9&0.691\\
\texttt{deberta-selfchecknli}&75.3&0.683&75.9&0.683\\
\texttt{t5\_xxl\_true\_nli\_mixture}&81.1&0.650&79.6&0.626\\
\hline
\texttt{baseline} & 69.7 & 0.403 & 74.5 & 0.488 \\
Our system\textsubscript{submitted} & 82.1 & 0.752 & 79.9 & 0.713 \\
Our system\textsubscript{best} & 82.5 & 0.757 & 79.9 & \textbf{0.722} \\
Best leaderboard & \textbf{84.7} & \textbf{0.770} & \textbf{81.3} & 0.715 \\
\end{tabular}
\caption{The results of the accuracy and Spearman correlation metrics on the test sample for the model-agnostic and model-aware tracks.}
\label{results}
\end{table*}

Among NLI pre-trained models, T5 model significantly outperformed other NLI models. However, our ensemble approach using features from all NLI pre-trained models significantly outperformed T5 in terms of correlation with annotators' decisions. 

Our approach for model-agnostic case provided us with an accuracy of 82.1\% and Spearman correlation of 0.752. With this approach, our team achieved the 6th place out of 41 in the competition for model-agnostic track. Only two teams achieved a higher Spearman correlation.

The same approach was applied for the model-aware track and provided us with an accuracy of 79.9\%, which is the 8th place out of 38 in the competition. The value of Spearman correlation turned out to be 0.713.

% In addition, we analyzed feature importance of the trained models.

More detailed results of the competition can be found in~\citet{mickus-etal-2024-semeval}.

\section{Conclusion}

In this paper, we describe the ensemble system for hallucination detection by using transformer-based models. We present a simple, yet effective ensemble pipeline that provided us with results comparable with the best scores for the both tracks.

Future work might include thoughtful error analysis. Improved quality can be achieved by annotating unlabeled training set with LLMs~\cite{ostyakova-etal-2023-chatgpt}. In addition, a multilingual setup of NLI models can be used to develop multilingual hallucination detection system~\cite{chizhikova, squad}. The proposed approach can be used standalone or can be integrated into the
DeepPavlov framework~\cite{burtsev2018deeppavlov}.

% Bibliography entries for the entire Anthology, followed by custom entries
% \bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}
\end{document}