@inproceedings{vasselli-etal-2026-measuring,
    title = "Measuring Linguistic Competence of {LLM}s on Indigenous Languages of the {A}mericas",
    author = "Vasselli, Justin  and
      Mp, Arturo  and
      Hudi, Frederikus  and
      Sakajo, Haruki  and
      Watanabe, Taro",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 2: Short Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-short.21/",
    pages = "287--296",
    ISBN = "979-8-89176-381-4",
    abstract = "This paper presents an evaluation framework for probing large language models' linguistic knowledge of Indigenous languages of the Americas using zero- and few-shot prompting. The framework consists of three tasks: (1) language identification, (2) cloze completion of Spanish sentences supported by Indigenous-language translations, and (3) grammatical feature classification. We evaluate models from five major families (GPT, Gemini, DeepSeek, Qwen, and LLaMA) on 13 Indigenous languages, including Bribri, Guarani, and Nahuatl. The results show substantial variation across both languages and model families. While a small number of model-language combinations demonstrate consistently stronger performance across tasks, many others perform near chance, highlighting persistent gaps in current models' abilities on Indigenous languages."
}