Khushnur Binte Jahangir
2026
Can LLMs Solve My Grandma’s Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles
Nurul Labib Sayeedi | Md. Faiyaz Abdullah Sayeedi | Khushnur Binte Jahangir | Swakkhar Shatabda | Sarah Masud Preum
Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)
Nurul Labib Sayeedi | Md. Faiyaz Abdullah Sayeedi | Khushnur Binte Jahangir | Swakkhar Shatabda | Sarah Masud Preum
Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)
Large Language Models (LLMs) show impressive performance on many NLP benchmarks, yet their ability to reason in figurative, culturally grounded, and low-resource settings remains underexplored. We address this gap for Bangla by introducing BanglaRiddleEval, a benchmark of 1,244 traditional Bangla riddles instantiated across four tasks (4,976 riddle-task artifacts in total). Using an LLM-based pipeline, we generate Chain-of-Thought explanations, semantically coherent distractors, and fine-grained ambiguity annotations, and evaluate a diverse suite of open-source and closed-source models under different prompting strategies. Models achieve moderate semantic overlap on generative QA but low correctness, MCQ accuracy peaks at only about 56% versus an 83.3% human baseline, and ambiguity resolution ranges from roughly 26% to 68%, with high-quality explanations confined to the strongest models. These results show that current LLMs capture some cues needed for Bangla riddle reasoning but remain far from human-level performance, establishing BanglaRiddleEval as a challenging new benchmark for low-resource figurative reasoning. All data, code, and evaluation scripts are available on GitHub: https://anonymous.4open.science/r/BanglaRiddleEval.