Understanding Native Language Identification for Brazilian Indigenous Languages
Paulo Cavalin, Pedro Domingues, Julio Nogima, Claudio Pinhanez
Abstract
We investigate native language identification (LangID) for Brazilian Indigenous Languages (BILs), using the Bible as training data. Our research extends from previous work, by presenting two analyses on the generalization of Bible-based LangID in non-biblical data. First, with newly collected non-biblical datasets, we show that such a LangID can still provide quite reasonable accuracy in languages for which there are more established writing standards, such as Guarani Mbya and Kaigang, but there can be a quite drastic drop in accuracy depending on the language. Then, we applied the LangID on a large set of texts, about 13M sentences from the Portuguese Wikipedia, towards understanding the difficulty factors may come out of such task in practice. The main outcome is that the lack of handling other American indigenous languages can affect considerably the precision for BILs, suggesting the need of a joint effort with related languages from the Americas.- Anthology ID:
- 2023.americasnlp-1.3
- Volume:
- Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Manuel Mager, Abteen Ebrahimi, Arturo Oncevay, Enora Rice, Shruti Rijhwani, Alexis Palmer, Katharina Kann
- Venue:
- AmericasNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 12–18
- Language:
- URL:
- https://aclanthology.org/2023.americasnlp-1.3
- DOI:
- 10.18653/v1/2023.americasnlp-1.3
- Cite (ACL):
- Paulo Cavalin, Pedro Domingues, Julio Nogima, and Claudio Pinhanez. 2023. Understanding Native Language Identification for Brazilian Indigenous Languages. In Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP), pages 12–18, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Understanding Native Language Identification for Brazilian Indigenous Languages (Cavalin et al., AmericasNLP 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.americasnlp-1.3.pdf