Hunter Scheppat
2026
FormosanMT: A Multilingual Parallel Corpus of the Formosan Language Family
Hunter Scheppat | Joshua K. Hartshorne | Sema Koc | Éric Le Ferrand | Emily Prud'hommeaux
Proceedings of the Fifteenth Language Resources and Evaluation Conference
Hunter Scheppat | Joshua K. Hartshorne | Sema Koc | Éric Le Ferrand | Emily Prud'hommeaux
Proceedings of the Fifteenth Language Resources and Evaluation Conference
While the quality of machine translation (MT) between widely-spoken languages has improved dramatically in recent years, training robust MT systems for languages with fewer resources remains a challenge. Endangered languages, which often lack the speaker population and written tradition needed to create text resources, are at a particular disadvantage. Developing robust MT architectures for very low-resource settings is hampered by the lack of suitable parallel corpora. To address this challenge, we introduce FormosanMT, a set of MT-ready parallel corpora for the Formosan family of endangered languages indigenous to Taiwan. Together the corpora total nearly 500,000 Formosan-Mandarin and Formosan-English sentence pairs. We share scripts for extracting these corpora from public sources, along with customizable tools for filtering, normalizing, and partitioning the data. In addition, we provide a new tokenizer for Traditional Chinese writing compatible with the popular No Language Left Behind (NLLB) MT architecture, along with updated and improved code for fine-tuning NLLB for any low-resource language pair. Finally we distribute our fully trained NLLB and OpenNMT models for the Formosan languages to and from both Mandarin and English. In addition to serving as a valuable resource for the Formosan language speaker communities, our data, code, and models will be available to NLP researchers working on endangered and low-resource language MT.
2025
Integrating diverse corpora for training an endangered language machine translation system
Hunter Scheppat | Joshua Hartshorne | Dylan Leddy | Eric Le Ferrand | Emily Prudhommeaux
Proceedings of the Eight Workshop on the Use of Computational Methods in the Study of Endangered Languages
Hunter Scheppat | Joshua Hartshorne | Dylan Leddy | Eric Le Ferrand | Emily Prudhommeaux
Proceedings of the Eight Workshop on the Use of Computational Methods in the Study of Endangered Languages
Machine translation (MT) can be a useful technology for language documentation and for promoting language use in endangered language communities. Few endangered languages, however, have an existing parallel corpus large enough to train a reasonable MT model. In this paper, we re-purpose a wide range of diverse data sources containing Amis, English, and Mandarin text to serve as parallel corpora for training MT systems for Amis, one of the Indigenous languages of Taiwan. To supplement the small amount of Amis-English data, we produce synthetic Amis-English data by using a high quality MT system to generate English translations for the Mandarin side of the Amis-Mandarin corpus. Using two popular neural MT systems, OpenNMT and NLLB, we train models to translate between English and Amis, and Mandarin and Amis. We find that including synthetic data is helpful only when translating to English. In addition, we observe that neither MT architecture is consistently superior to other and that performance seems to vary according to the direction of translation and the amount of data used. These results indicate that MT is possible for an under-resourced language even without a formally prepared parallel corpus, but multiple training methods should be explored to produce optimal results.