@inproceedings{shahriar-etal-2025-p6jiggasha,
    title = "{P}6{J}iggasha: Benchmarking Large Language Models on {B}angla Physics Question Answering with Cross-lingual Evaluation",
    author = "Shahriar, S.m.  and
      Fuad, Md Tahmid Hasan  and
      Fahim, Md  and
      Hossain, Md. Azad",
    editor = "Alam, Firoj  and
      Kar, Sudipta  and
      Chowdhury, Shammur Absar  and
      Hassan, Naeemul  and
      Prince, Enamul Hoque  and
      Tasnim, Mohiuddin  and
      Rony, Md Rashad Al Hasan  and
      Rahman, Md Tahmid Rahman",
    booktitle = "Proceedings of the Second Workshop on Bangla Language Processing (BLP-2025)",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.banglalp-1.16/",
    pages = "196--211",
    ISBN = "979-8-89176-314-2",
    abstract = "Understanding scientific concepts in native languages is crucial for educational accessibility and knowledge transfer. In this work, we present a comprehensive evaluation of Large Language Models (LLMs) on Bangla physics questions, introducing P6Jiggasha, a novel dataset of 1,500 multiple-choice questions compiled from HSC physics textbooks, supplementary guides, admission preparation books, and past examination papers from various educational boards. We evaluate three state-of-the-art models{---}GPT-4.1, Gemini-2.5 Pro, and DeepSeek-R1-Distill-Llama-70B{---}on both native Bangla questions and their English translations. Our results reveal significant performance variations, with GPT-4.1 achieving 86.67{\%} accuracy on Bangla questions in a single inference, while other models show substantial improvement through multiple inference attempts, with Gemini-2.5 Pro reaching 89.52{\%} after four iterations. We introduce a \textit{Cumulative Accuracy@k} metric to evaluate iterative reasoning capabilities and provide comprehensive analysis across six physics topics and six question types. Our error analysis reveals systematic cross-lingual inconsistencies where models produce contradictory answers for identical questions across languages. This study provides valuable insights into the capabilities and limitations of current LLMs for low-resource scientific question answering and establishes benchmarks for future research in Bangla natural language processing."
}