@inproceedings{mainzinger-brixey-2026-indigieval,
    title = "{I}ndigi{E}val: Evaluating {LLM}s in {N}orth {A}merican Indigenous Languages",
    author = "Mainzinger, Julia  and
      Brixey, Jacqueline",
    editor = "Mager, Manuel  and
      Ebrahimi, Abteen  and
      Bui, Minh Duc  and
      Pugh, Robert  and
      Oncevay, Arturo  and
      Chiruzzo, Luis  and
      Solano, Rolando Coto  and
      Rijhwani, Shruti  and
      Von Der Wense, Katharina",
    booktitle = "Proceedings of the Sixth Workshop on {NLP} for Indigenous Languages of the {A}mericas ({A}mericas{NLP})",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.americasnlp-6.8/",
    pages = "82--94",
    ISBN = "979-8-89176-415-6",
    abstract = "This paper presents IndigiEval, a framework for evaluating the language and cultural proficiency of several commercially available large language models (LLMs) across five North American Indigenous languages (Mvskoke, Choctaw, Cherokee, Cheyenne, and Hawaiian). This framework is a qualitative evaluation method intended for communities with small speaker populations to be able to critically evaluate LLM performance with minimal data and human effort. IndigiEval includes tasks such as answering cultural questions, translation, text generation, and speech recognition. The results of our experiments indicate that no currently available LLM performs well across all evaluation categories, and that LLMs frequently hallucinate orthographies, grammatical structures, cultural knowledge, and vocabulary for all languages and cultures considered. Our proposed evaluation framework is not intended as a comprehensive score, but rather a qualitative and flexible framework to inform language communities about a given LLM{'}s potential as a resource, since each language has unique environments, strengths, and availability of resources."
}