SubmissionNumber#=%=#205 FinalPaperTitle#=%=#MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thought Prompts ShortPaperTitle#=%=# NumberOfPages#=%=#6 CopyrightSigned#=%=#Md Nishat Raihan JobTitle#==# Organization#==#George Mason University, 4400 University Dr, Fairfax, Virginia 22030, USA. Abstract#==#Our paper presents team MasonTigers submission to the SemEval-2024 Task 9 - which provides a dataset of puzzles for testing natural language understanding. We employ large language models (LLMs) to solve this task through several prompting techniques. Zero-shot and few-shot prompting generate reasonably good results when tested with proprietary LLMs, compared to the open-source models. We obtain further improved results with chain-of-thought prompting, an iterative prompting method that breaks down the reasoning process step-by-step. We obtain our best results by utilizing an ensemble of chain-of-thought prompts, placing 2nd in the word puzzle subtask and 13th in the sentence puzzle subtask. The strong performance of prompted LLMs demonstrates their capability for complex reasoning when provided with a decomposition of the thought process. Our work sheds light on how step-wise explanatory prompts can unlock more of the knowledge encoded in the parameters of large models. Author{1}{Firstname}#=%=#Nishat Author{1}{Lastname}#=%=#Raihan Author{1}{Username}#=%=#md.nishat Author{1}{Email}#=%=#mraihan2@gmu.edu Author{1}{Affiliation}#=%=#George Mason University Author{2}{Firstname}#=%=#Dhiman Author{2}{Lastname}#=%=#Goswami Author{2}{Username}#=%=#dhimangoswami Author{2}{Email}#=%=#dgoswam@gmu.edu Author{2}{Affiliation}#=%=#George Mason University Author{3}{Firstname}#=%=#Al Nahian Author{3}{Lastname}#=%=#Bin Emran Author{3}{Username}#=%=#alnahian Author{3}{Email}#=%=#abinemra@gmu.edu Author{3}{Affiliation}#=%=#George Mason University Author{4}{Firstname}#=%=#Sadiya Sayara Chowdhury Author{4}{Lastname}#=%=#Puspo Author{4}{Username}#=%=#sadiyapuspo Author{4}{Email}#=%=#spuspo@gmu.edu Author{4}{Affiliation}#=%=#George Mason University Author{5}{Firstname}#=%=#Amrita Author{5}{Lastname}#=%=#Ganguly Author{5}{Username}#=%=#amritaganguly Author{5}{Email}#=%=#agangul@gmu.edu Author{5}{Affiliation}#=%=#George Mason University Author{6}{Firstname}#=%=#Marcos Author{6}{Lastname}#=%=#Zampieri Author{6}{Username}#=%=#marcoszampieri Author{6}{Email}#=%=#mzampier@gmu.edu Author{6}{Affiliation}#=%=#George Mason University ========== èéáğö