@article{p-m-etal-2026-riddle,
    title = "The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual {LLM}s Using {I}ndian Riddles",
    author = "P M, Abhinav  and
      Saxena, Ojasva  and
      C, Oswald  and
      Krishnamurthy, Parameswari",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    journal = "International Conference on Language Resources and Evaluation",
    volume = "main",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.lrec-main.430/",
    pages = "5516--5527",
    abstract = "The extent to which large language models (LLMs) can perform culturally grounded reasoning across non-English languages remains underexplored. This paper examines the reasoning and self-assessment abilities of LLMs across seven major Indian languages- Bengali, Gujarati, Hindi, Kannada, Malayalam, Tamil, and Telugu. We introduce a multilingual riddle dataset combining traditional riddles with context-reconstructed variants and evaluate five LLMs- Gemini 2.5 Pro, Gemini 2.5 Flash, Mistral-Saba, LLaMA-4-Scout, and LLaMA-4-Maverick under seven prompting strategies. In the first stage, we assess riddle-solving performance and find that while Gemini 2.5 Pro performs best overall, few-shot methods yield only marginal gains, and accuracy varies notably across languages. In the second stage, we conduct a self-evaluation experiment to measure reasoning consistency. The results reveal a key finding: a model{'}s initial accuracy is inversely correlated with its ability to identify its own mistakes. Top-performing models such as Gemini 2.5 Pro are overconfident (4.34{\%} True Negative Rate), whereas lower-performing models like LLaMA-4-Scout are substantially more self-aware (42.09{\%} True Negative Rate). These results point to clear gaps in multilingual reasoning and highlight the need for models that not only reason effectively but also recognize their own limitations."
}