\section{Introduction}

Progress in the field of automatic speech recognition (ASR) for high-resource languages has led to several large multilingual speech models with human-level performance on popular benchmarks \citep{meta_omnilingual_asr_blog, mms_paper, whisper_l_v3_paper, xls_r_paper, yadav-sitaram-2022-survey}. Of equal importance is the need to model the linguistic diversity in the majority of the 7,000+ languages in the world that are either endangered, on the brink of extinction, low-resource, or have few native speakers left.

Building speech tools (such as ASR models) for indigenous languages has applications in language documentation \citep{asr_end_lng_doc, shi-etal-2021-leveraging, jimerson-prudhommeaux-2018-asr}, language revitalization \citep{mainzinger-2024-technology, zhang-etal-2022-nlp}, community language learning \citep{asr_lg_learn, dolinska-etal-2024-akha, sec_lg_learning, sun_lg_learn, lang_learn_journal}, and user accessibility \citep{asr_accessibility_1, asr_accessibility_2, asr_accessibility_3, asr_accessibility_4}. Not only can ASR tools help generate valuable synthetic data to help augment languages with scarce resources, leading to higher resources \citep{venkateswaran-liu-2024-looking, machine_chain}, they can also be used to assist linguists and fieldworkers in improving transcription error rates and time-to-transcribe when building gold standard corpora \citep{prud2021automatic} for indigenous languages. 


These are not standalone issues. The use of NLP tools in language preservation ensures that seminal information about cultural artefacts can be passed down to future generations \citep{preserve_asr_1, preserve_asr_2, preserve_asr_3, preserve_asr_4}.

In this work, we fine tune and evaluate popular multilingual speech models on 21 low-resource languages and employ $n$-gram modeling to enhance decoding accuracy. Our empirical analysis sheds light on the efficacy of using these models for building ASR tools and highlights significant gaps in model performance, thereby justifying the need to create more resources and build corpora for underserved languages worldwide. 


Our paper has three contributions. First, we provide a detailed background of two languages in our dataset - Scots (sco) and Nubi (kcn) in order to highlight their history, syntax, morphology, lack of labeled speech corpora, and difficulty in modeling speech tools (\S\ref{sec:bg}, \S\ref{sec:related}). Second, we provide a comprehensive review of error rates after fine-tuning three popular multilingual models and compare their performance. We augment beam search with an external $n$-gram language model estimated from training text, and combine acoustic and LM scores via shallow fusion(\S\ref{sec:methods}). Finally, we perform an extensive linguistic-error analysis of the transcripts generated for the test set by our best-performing models on Scots and Nubi and highlight key challenges and problems that occur with our fine-tuned models (\S\ref{sec:lg}). 

Our choice of Scots and Nubi for linguistic error analysis is motivated by two reasons: (1) both share a ``shadow'' relationship with a high-resource language (English for Scots, Arabic for Nubi), making them a natural minimal pair for asking whether genetic/lexical proximity to a well-modeled language helps ASR, and (2) all authors are native English speakers, which enabled meaningful analysis of Scots transcripts at the lexical and phonological level.

In the next section (\S\ref{sec:data}), we provide a brief overview of our dataset and some initial pre-processing steps taken to clean and split it.