@inproceedings{kamath-etal-2025-benchmarking,
    title = "Benchmarking {H}indi {LLM}s: A New Suite of Datasets and a Comparative Analysis",
    author = "Kamath, Anusha  and
      Singla, Kanishk  and
      Paul, Rakesh  and
      Joshi, Raviraj Bhuminand  and
      Vaidya, Utkarsh  and
      Chauhan, Sanjay Singh  and
      Wartikar, Niranjan",
    editor = "Bhattacharya, Arnab  and
      Goyal, Pawan  and
      Ghosh, Saptarshi  and
      Ghosh, Kripabandhu",
    booktitle = "Proceedings of the 1st Workshop on Benchmarks, Harmonization, Annotation, and Standardization for Human-Centric AI in Indian Languages (BHASHA 2025)",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.bhasha-1.5/",
    pages = "52--68",
    ISBN = "979-8-89176-313-5",
    abstract = "Evaluating instruction-tuned Large Language Models (LLMs) in Hindi is challenging due to a lack of high-quality benchmarks, as direct translation of English datasets fails to capture crucial linguistic and cultural nuances. To address this, we introduce a suite of five Hindi LLM evaluation datasets: IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, and BFCL-Hi. These were created using a methodology that combines from-scratch human annotation with a translate-and-verify process. We leverage this suite to conduct an extensive benchmarking of open-source LLMs supporting Hindi, providing a detailed comparative analysis of their current capabilities. Our curation process also serves as a replicable methodology for developing benchmarks in other low-resource languages."
}