SubmissionNumber#=%=#175
FinalPaperTitle#=%=#SLPL SHROOM at SemEval2024 Task 06 : A comprehensive study on models ability to detect hallucination
ShortPaperTitle#=%=#
NumberOfPages#=%=#7
CopyrightSigned#=%=#Pouya Fallah
JobTitle#==#Student
Organization#==#
Abstract#==#Language models, particularly generative models, are susceptible to hallucinations, generating outputs that contradict factual knowledge
or the source text. This study explores methods
for detecting hallucinations in three SemEval2024 Task 6 tasks: Machine Translation, Definition Modeling, and Paraphrase Generation.
We evaluate two methods: semantic similarity
between the generated text and factual references, and an ensemble of language models
that judge each other's outputs. Our results
show that semantic similarity achieves moderate accuracy and correlation scores in trial data,
while the ensemble method offers insights into
the complexities of hallucination detection but
falls short of expectations. This work highlights the challenges of hallucination detection
and underscores the need for further research
in this critical area.
Author{1}{Firstname}#=%=#Pouya
Author{1}{Lastname}#=%=#Fallah
Author{1}{Username}#=%=#pooya-fallah
Author{1}{Email}#=%=#pooya.fallah79@sharif.edu
Author{1}{Affiliation}#=%=#Sharif University of Technology
Author{2}{Firstname}#=%=#Soroush
Author{2}{Lastname}#=%=#Gooran
Author{2}{Email}#=%=#gooran@sharif.edu
Author{2}{Affiliation}#=%=#Sharif University of Technology
Author{3}{Firstname}#=%=#Mohammad
Author{3}{Lastname}#=%=#Jafarinasab
Author{3}{Email}#=%=#mohammad_jafari_nasab@yahoo.com
Author{3}{Affiliation}#=%=#Sharif University of Technology
Author{4}{Firstname}#=%=#Pouya
Author{4}{Lastname}#=%=#Sadeghi
Author{4}{Email}#=%=#pouya.sadeghi@ut.ac.ir
Author{4}{Affiliation}#=%=#University of Tehran
Author{5}{Firstname}#=%=#Reza
Author{5}{Lastname}#=%=#Farnia
Author{5}{Email}#=%=#r.farnia0212@sharif.edu
Author{5}{Affiliation}#=%=#Sharif University of Technology
Author{6}{Firstname}#=%=#Amirreza
Author{6}{Lastname}#=%=#Tarabkhah
Author{6}{Email}#=%=#tarabkhah2@aut.ac.ir
Author{6}{Affiliation}#=%=#Amirkabir University of Technology
Author{7}{Firstname}#=%=#Zeinab Sadat
Author{7}{Lastname}#=%=#Taghavi
Author{7}{Email}#=%=#zeinabtaghavi@sharif.edu
Author{7}{Affiliation}#=%=#Sharif University of Technology
Author{8}{Firstname}#=%=#Hossein
Author{8}{Lastname}#=%=#Sameti
Author{8}{Email}#=%=#sameti@sharif.edu
Author{8}{Affiliation}#=%=#Sharif University of Technology

==========
èéáğö