@inproceedings{singh-etal-2025-indic,
    title = "{INDIC} {QA} {BENCHMARK}: A Multilingual Benchmark to Evaluate Question Answering capability of {LLM}s for {I}ndic Languages",
    author = "Singh, Abhishek Kumar  and
      Kumar, Vishwajeet  and
      Murthy, Rudra  and
      Sen, Jaydeep  and
      Mittal, Ashish  and
      Ramakrishnan, Ganesh",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.141/",
    pages = "2607--2626",
    ISBN = "979-8-89176-195-7",
    abstract = "Large Language Models (LLMs) perform well on unseen tasks in English, but their abilities in non-English languages are less explored due to limited benchmarks and training data. To bridge this gap, we introduce the Indic-QA Benchmark, a large dataset for context-grounded question answering in 11 major Indian languages, covering both extractive and abstractive tasks. Evaluations of multilingual LLMs, including instruction fine-tuned versions, revealed weak performance in low-resource languages due to a strong English-language bias in their training data. We also investigated the Translate-Test paradigm,where inputs are translated to English for processing and the results are translated back into the source language for output. This approach outperformed multilingual LLMs, particularly in low-resource settings. By releasing Indic-QA, we aim to promote further research into LLMs' question-answering capabilities in low-resource languages. This benchmark offers a critical resource to address existing limitations and foster multilingual understanding."
}