SubmissionNumber#=%=#297 FinalPaperTitle#=%=#SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes ShortPaperTitle#=%=# NumberOfPages#=%=#15 CopyrightSigned#=%=#Timothee Mickus JobTitle#==# Organization#==# Abstract#==#This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet inaccurate. Such cases of overgeneration put in jeopardy many NLG applications, where correctness is often mission-critical. The shared task was conducted with a newly constructed dataset of 4000 model outputs labeled by 5 annotators each, spanning 3 NLP tasks: machine translation, paraphrase generation and definition modeling. The shared task was tackled by a total of 58 different users grouped in 42 teams, out of which 26 elected to write a system description paper; collectively, they submitted over 300 prediction sets on both tracks of the shared task. We observe a number of key trends in how this approach was tackled—many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items. Author{1}{Firstname}#=%=#Timothee Author{1}{Lastname}#=%=#Mickus Author{1}{Username}#=%=#tmickus Author{1}{Email}#=%=#timothee.mickus@helsinki.fi Author{1}{Affiliation}#=%=#University of Helsinki Author{2}{Firstname}#=%=#Elaine Author{2}{Lastname}#=%=#Zosa Author{2}{Username}#=%=#elainezosa Author{2}{Email}#=%=#elaine.zosa@silo.ai Author{2}{Affiliation}#=%=#SiloGen Author{3}{Firstname}#=%=#Raul Author{3}{Lastname}#=%=#Vazquez Author{3}{Username}#=%=#raul.vazquez Author{3}{Email}#=%=#raul.vazquez@helsinki.fi Author{3}{Affiliation}#=%=#University of Helsinki Author{4}{Firstname}#=%=#Teemu Author{4}{Lastname}#=%=#Vahtola Author{4}{Username}#=%=#texvahto Author{4}{Email}#=%=#teemu.vahtola@helsinki.fi Author{4}{Affiliation}#=%=#University of Helsinki Author{5}{Firstname}#=%=#Jörg Author{5}{Lastname}#=%=#Tiedemann Author{5}{Username}#=%=#joerg Author{5}{Email}#=%=#jorg.tiedemann@helsinki.fi Author{5}{Affiliation}#=%=#University of Helsinki Author{6}{Firstname}#=%=#Vincent Author{6}{Lastname}#=%=#Segonne Author{6}{Username}#=%=#vsegonne Author{6}{Email}#=%=#vincent.segonne@univ-ubs.fr Author{6}{Affiliation}#=%=#IRISA - Université Bretagne Sud Author{7}{Firstname}#=%=#Alessandro Author{7}{Lastname}#=%=#Raganato Author{7}{Username}#=%=#alessandro.raganato Author{7}{Email}#=%=#alessandro.raganato@unimib.it Author{7}{Affiliation}#=%=#University of Milano-Bicocca Author{8}{Firstname}#=%=#Marianna Author{8}{Lastname}#=%=#Apidianaki Author{8}{Username}#=%=#marianna.apidianaki Author{8}{Email}#=%=#marapi@seas.upenn.edu Author{8}{Affiliation}#=%=#University of Pennsylvania ========== èéáğö