SubmissionNumber#=%=#284
FinalPaperTitle#=%=#Zero Shot is All You Need at SemEval-2024 Task 9: A study of State of the Art LLMs on Lateral Thinking Puzzles
ShortPaperTitle#=%=#
NumberOfPages#=%=#5
CopyrightSigned#=%=#Erfan Moosavi Monazzah
JobTitle#==#
Organization#==#
Abstract#==#The successful deployment of large language models in numerous NLP tasks has spurred the demand for tackling more complex tasks, which were previously unattainable. SemEval-2024 Task 9 introduces the brainteaser dataset that necessitates intricate, human-like reasoning to solve puzzles that challenge common sense. At first glance, the riddles in the dataset may appear trivial for humans to solve. However, these riddles demand lateral thinking, which deviates from vertical thinking that is the dominant form when it comes to current reasoning tasks. In this paper, we examine the ability of current state-of-the-art LLMs to solve this task. Our study is diversified by selecting both open and closed source LLMs with varying numbers of parameters. Additionally, we extend the task dataset with synthetic explanations derived from the LLMs' reasoning processes during task resolution. These could serve as a valuable resource for further expanding the task dataset and developing more robust methods for tasks that require complex reasoning. All the codes and datasets are available in paper's GitHub repository.
Author{1}{Firstname}#=%=#Erfan
Author{1}{Lastname}#=%=#Moosavi Monazzah
Author{1}{Username}#=%=#erfanmoosavimonazzah
Author{1}{Email}#=%=#erfan.moosavi.monazzah@gmail.com
Author{1}{Affiliation}#=%=#Iran University of Science and Technology
Author{2}{Firstname}#=%=#Mahdi
Author{2}{Lastname}#=%=#Feghhi
Author{2}{Email}#=%=#feghhi_me@comp.iust.ac.ir
Author{2}{Affiliation}#=%=#Iran University of Science and Technology

==========
èéáğö