@inproceedings{hashiloni-etal-2025-dharmabench,
    title = "{D}harma{B}ench: Evaluating Language Models on Buddhist Texts in {S}anskrit and {T}ibetan",
    author = "Hashiloni, Kai Golan  and
      Cohen, Shay  and
      Shina, Asaf  and
      Yang, Jingyi  and
      Zwebner, Orr Meir  and
      Bajetta, Nicola  and
      Bilitski, Guy  and
      Sund{\'e}n, Rebecca  and
      Maduel, Guy  and
      Conlon, Ryan  and
      Barzilai, Ari  and
      Mass, Daniel  and
      Jia, Shanshan  and
      Naaman, Aviv  and
      Choden, Sonam  and
      Jamtsho, Sonam  and
      Qu, Yadi  and
      Isaacson, Harunaga  and
      Wangchuk, Dorji  and
      Fine, Shai  and
      Almogi, Orna  and
      Bar, Kfir",
    editor = "Inui, Kentaro  and
      Sakti, Sakriani  and
      Wang, Haofen  and
      Wong, Derek F.  and
      Bhattacharyya, Pushpak  and
      Banerjee, Biplab  and
      Ekbal, Asif  and
      Chakraborty, Tanmoy  and
      Singh, Dhirendra Pratap",
    booktitle = "Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "The Asian Federation of Natural Language Processing and The Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.ijcnlp-long.114/",
    pages = "2088--2110",
    ISBN = "979-8-89176-298-5",
    abstract = "We assess the capabilities of large language models on tasks involving Buddhist texts written in Sanskrit and Classical Tibetan{---}two typologically distinct, low-resource historical languages. To this end, we introduce DharmaBench, a benchmark suite comprising 13 classification and detection tasks grounded in Buddhist textual traditions: six in Sanskrit and seven in Tibetan, with four shared across both. The tasks are curated from scratch, tailored to the linguistic and cultural characteristics of each language. We evaluate a range of models, from proprietary systems like GPT-4o to smaller, domain-specific open-weight models, analyzing their performance across tasks and languages. All datasets and code are publicly released, under the CC-BY-4 License and the Apache-2.0 License respectively, to support research on historical language processing and the development of culturally inclusive NLP systems."
}