\section{Results}

We present the results of our fine-tuning experiments in Table \ref{tab:asr-results}. The results are separated by continent of origin. For MMS and XLS-R, we collected \texttt{WER} and \texttt{CER} with and without unigram Language Model (LM) decoding. The bolded and underlined numbers are the best \texttt{WER} and \texttt{CER} for a given language, respectively. Interestingly, MMS achieves the lowest \texttt{WER} amongst all three models for 9 out of the 21 languages. Whisper achieves the lowest \texttt{WER} for 8 of 21 languages, whereas XLS-R only achieves this for 4 languages. Similarly, MMS achieves the lowest \texttt{CER} for 13 of the 21 languages. 6 languages achieve their lowest \texttt{CER} with XLS-R, whereas only 2 do so with Whisper. 

LM decoding consistently improves model performance. When fine-tuning MMS, using LM decoding reduces the \texttt{WER} of 17 languages, whereas with XLS-R, LM decoding helps reduce the \texttt{WER} of 19 languages. For languages where the use of LM decoding worsened the \texttt{WER}, the increase is not more than \verb|1%| absolute \texttt{WER}. These include Chiga, Konzo, Kenyi, Michoacán Mazahua, and Cypriot Greek. An exception to this trend is Papantla Totonac.

Of all languages, MMS achieves the best performance on Rutoro (\texttt{WER} = 0.241 and \texttt{CER} = 0.043), XLS-R achieves the best performance on Scots (\texttt{WER} = 0.297 and \texttt{CER} = 0.113), and Whisper performs best on Western Penan (\texttt{WER} = 0.264 and \texttt{CER} = 0.113).

\subsection{How Much Data is Enough Data?}

We attempt to ablate the number of training hours to capture model performance deterioration. In Table \ref{tab:nubi-scots-wer} we evaluate the models on two training splits of Scots and Nubi - a 1 hr training data split and a 50\% split (3 hrs for Nubi and 5 hrs for Scots). We observe that Whisper outperforms other models in extremely low-resource scenarios by significant margins. Whisper's low \texttt{WER} on just 1 hr of Scots is evidence that model performance in a given language is directly correlated with the amount of training data in that language (Whisper is trained on $438k+$ hrs of English audio alone).

\subsection{Effect of $n$-gram Language Decoding}

To improve CTC decoding beyond unigram baselines, we augment CTC beam search with an external $n$-gram language model estimated from training text, combining acoustic and LM scores via shallow fusion (weighted by $\alpha$ and $\beta$). We sweep $n$ $\in$ \{3,4\}, $\alpha$, $\beta$, and beam width across all 21 languages for the models. Table \ref{tab:lm-xlsr-small} provides \texttt{WER} for select languages for the best-performing sweep parameters, and Table \ref{tab:lm-ablation-full} in Appendix \ref{appendixD} gives full sweep results across all models and settings.

We observe that 4-gram LM decoding consistently outperforms greedy and unigram decoding for most languages, with improvements up to $27.3\%$ in \texttt{WER}. Improvements are geographically broad, while only a few languages show mild degradation under specific settings overall.

\subsection{What Drives Cross-Model Variance?}
\label{sec:attr}

A natural question is whether the observed performance gaps reflect architectural differences, language properties, or our fine-tuning and decoding setup. While we cannot fully disentangle these factors, three patterns in our results emerge. First, Whisper's advantage is concentrated in languages where either the target language or a closely related high-resource language is well represented in its pretraining data: it achieves the lowest WER on Scots in the 1-hour setting (Table \ref{tab:nubi-scots-wer}), Western Penan, Gheg Albanian, and Cypriot Greek — all languages with substantial English, Indonesian/Malay, or Greek pretraining exposure to draw on. Second, MMS performs best on languages with simpler orthographies and character inventories well-covered by its 1162-language pretraining (e.g., Rutoro, Chiga, Bukusu), where its language-specific adapters appear to give it a head start over architectures without per-language conditioning. Third, the largest gains from n-gram LM decoding (Table 6) accrue to XLS-R rather than MMS, suggesting that XLS-R's CTC outputs are more under-constrained at the lexical level and benefit disproportionately from external lexical priors. Architectural and pretraining-distribution effects, therefore, appear intertwined; isolating them would require controlled pretraining ablations beyond the scope of this work.
