SubmissionNumber#=%=#36
FinalPaperTitle#=%=#BAMO at SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense
ShortPaperTitle#=%=#
NumberOfPages#=%=#9
CopyrightSigned#=%=#Baktash Ansari
JobTitle#==#
Organization#==#Name: Iran University of Science and Technology (IUST).
Address: University of Science and Technology of Iran, University St., Hengam St., Resalat Square, Tehran, Iran.
Abstract#==#This paper outlines our approach to SemEval 2024 Task 9, BRAINTEASER: A Novel Task Defying Common Sense. The task aims to evaluate the ability of language models to think creatively. The dataset comprises multi-choice questions that challenge models to think 'outside of the box'. We fine-tune 2 models, BERT and RoBERTa Large. Next, we employ a Chain of Thought (CoT) zero-shot prompting approach with 6 large language models, such as GPT-3.5, Mixtral, and Llama2. Finally, we utilize ReConcile, a technique that employs a ‘round table conference' approach with multiple agents for zero-shot learning, to generate consensus answers among 3 selected language models. Our best method achieves an overall accuracy of 85 percent on the sentence puzzles subtask.
Author{1}{Firstname}#=%=#Baktash
Author{1}{Lastname}#=%=#Ansari
Author{1}{Username}#=%=#baktash
Author{1}{Email}#=%=#baktash.ansari1381@gmail.com
Author{1}{Affiliation}#=%=#Iran University of Science and Technology
Author{2}{Firstname}#=%=#Mohammadmostafa
Author{2}{Lastname}#=%=#Rostamkhani
Author{2}{Username}#=%=#mohammadmostafa
Author{2}{Email}#=%=#mohammadmostafarostamkhani@gmail.com
Author{2}{Affiliation}#=%=#IUST
Author{3}{Firstname}#=%=#Sauleh
Author{3}{Lastname}#=%=#Eetemadi
Author{3}{Username}#=%=#sauleh
Author{3}{Email}#=%=#sauleh@iust.ac.ir
Author{3}{Affiliation}#=%=#Iran University of Science and Technology

==========
èéáğö