SubmissionNumber#=%=#151 FinalPaperTitle#=%=#MAINDZ at SemEval-2024 Task 5: CLUEDO - Choosing Legal oUtcome by Explaining Decision through Oversight ShortPaperTitle#=%=# NumberOfPages#=%=#9 CopyrightSigned#=%=#Irene Benedetto JobTitle#==# Organization#==#Politenico di Torino, Corso Castelfidardo, 39, 10129 Torino TO MAIZE SRL, Via San Quintino 33, 10129 Torino TO Abstract#==#Large language models (LLMs) have recently obtained strong performance on complex reasoning tasks. However, their capabilities in specialized domains like law remain relatively unexplored. We present CLUEDO, a system to tackle a novel legal reasoning task that involves determining if a provided answer correctly addresses a legal question derived from U.S. civil procedure cases. CLUEDO utilizes multiple collaborator models that are trained using multiple-choice prompting to choose the right label and generate explanations. These collaborators are overseen by a final "detective" model that identifies the most accurate answer in a zero-shot manner. Our approach achieves an F1 macro score of 0.74 on the development set and 0.76 on the test set, outperforming individual models. Unlike the powerful GPT-4, CLUEDO provides more stable predictions thanks to the ensemble approach. Our results showcase the promise of tailored frameworks to enhance legal reasoning capabilities in LLMs. Author{1}{Firstname}#=%=#Irene Author{1}{Lastname}#=%=#Benedetto Author{1}{Username}#=%=#irene.benedetto Author{1}{Email}#=%=#irene.benedetto@polito.it Author{1}{Affiliation}#=%=#Politecnico di Torino Author{2}{Firstname}#=%=#Alkis Author{2}{Lastname}#=%=#Koudounas Author{2}{Username}#=%=#alkiskoudounas Author{2}{Email}#=%=#alkis.koudounas@polito.it Author{2}{Affiliation}#=%=#Politecnico di Torino Author{3}{Firstname}#=%=#Lorenzo Author{3}{Lastname}#=%=#Vaiani Author{3}{Username}#=%=#lvaiani Author{3}{Email}#=%=#lorenzo.vaiani@polito.it Author{3}{Affiliation}#=%=#Politecnico di Torino Author{4}{Firstname}#=%=#Eliana Author{4}{Lastname}#=%=#Pastor Author{4}{Username}#=%=#eliana.pastor Author{4}{Email}#=%=#eliana.pastor@polito.it Author{4}{Affiliation}#=%=#Politecnico di Torino Author{5}{Firstname}#=%=#Luca Author{5}{Lastname}#=%=#Cagliero Author{5}{Username}#=%=#luca.cagliero Author{5}{Email}#=%=#luca.cagliero@polito.it Author{5}{Affiliation}#=%=#Politecnico di Torino Author{6}{Firstname}#=%=#Francesco Author{6}{Lastname}#=%=#Tarasconi Author{6}{Email}#=%=#francesco.tarasconi@maize.io Author{6}{Affiliation}#=%=#MAIZE SRL ========== èéáğö