@inproceedings{maji-etal-2025-sanskriti,
    title = "{SANSKRITI}: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of {I}ndian Culture",
    author = "Maji, Arijit  and
      Kumar, Raghvendra  and
      Ghosh, Akash  and
      Anushka, Anushka  and
      Saha, Sriparna",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.228/",
    pages = "4434--4451",
    ISBN = "979-8-89176-256-5",
    abstract = "Language models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce \textit{ \textbf{SANSKRITI}}, a benchmark designed to evaluate language models' comprehension of India{'}s rich cultural diversity. Comprising of 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, \textit{ \textbf{SANSKRITI}} is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture namely rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife and personalities, providing a comprehensive representation of India{'}s cultural tapestry. We evaluate \textit{ \textbf{SANSKRITI}} on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models(SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, \textit{ \textbf{SANSKRITI}} sets a new standard for assessing and improving the cultural understanding of LMs. We will share the dataset and findings publicly to support research on inclusive and culturally aware AI systems."
}