SubmissionNumber#=%=#182 FinalPaperTitle#=%=#TU Wien at SemEval-2024 Task 6: Unifying Model-Agnostic and Model-Aware Techniques for Hallucination Detection ShortPaperTitle#=%=# NumberOfPages#=%=#14 CopyrightSigned#=%=#Varvara Arzt JobTitle#==# Organization#==#Faculty of Informatics, TU Wien; Favoritenstraße 9/11, 1040 Wien, Austria Abstract#==#This paper discusses challenges in Natural Language Generation (NLG), specifically addressing neural networks producing output that is fluent but incorrect, leading to "hallucinations". The SHROOM shared task involves Large Language Models in various tasks, and our methodology employs both model-agnostic and model-aware approaches for hallucination detection. The limited availability of labeled training data is addressed through automatic label generation strategies. Model-agnostic methods include word alignment and fine-tuning a BERT-based pretrained model, while model-aware methods leverage separate classifiers trained on LLMs' internal data (layer activations and attention values). Ensemble methods combine outputs through various techniques such as regression metamodels, voting, and probability fusion. Our best performing systems achieved an accuracy of 80.6% on the model-aware track and 81.7% on the model-agnostic track, ranking 3rd and 8th among all systems, respectively. Author{1}{Firstname}#=%=#Varvara Author{1}{Lastname}#=%=#Arzt Author{1}{Username}#=%=#varvara_arzt Author{1}{Email}#=%=#varvara.arzt@tuwien.ac.at Author{1}{Affiliation}#=%=#Vienna University of Technology Author{2}{Firstname}#=%=#Mohammad Mahdi Author{2}{Lastname}#=%=#Azarbeik Author{2}{Email}#=%=#mohammad.azarbeik@tuwien.ac.at Author{2}{Affiliation}#=%=#Vienna University of Technology Author{3}{Firstname}#=%=#Ilya Author{3}{Lastname}#=%=#Lasy Author{3}{Email}#=%=#ilya.lasy@tuwien.ac.at Author{3}{Affiliation}#=%=#Vienna University of Technology Author{4}{Firstname}#=%=#Tilman Author{4}{Lastname}#=%=#Kerl Author{4}{Email}#=%=#tilman.kerl@tuwien.ac.at Author{4}{Affiliation}#=%=#Vienna University of Technology Author{5}{Firstname}#=%=#Gábor Author{5}{Lastname}#=%=#Recski Author{5}{Email}#=%=#gabor.recski@tuwien.ac.at Author{5}{Affiliation}#=%=#Vienna University of Technology ========== èéáğö