We experiment with a number of settings to evaluate the performance of popular ASR systems on languages with varying linguistic features and available labeled speech data.

\subsection{Models}
For fine-tuning on all 21 languages in our dataset, we consider three models - Massively Multilingual Speech (MMS) \texttt{(facebook/mms-1b-all)}, XLS-R (\verb|facebook/wav2vec2-xls-r-1b|), and Whisper (\verb|openai/whisper-large-v3|). The choice of models for this experiment is driven by two factors: (1) all three models are trained on multilingual datasets, and (2) the models have a comparable number of trainable parameters - $1$B for MMS and XLS-R and $1.5$B for Whisper, making the performance comparison fair.

The Massively Multilingual Project \citep{mms_paper} is based on the Wav2Vec 2.0 architecture \citep{wav_2_vec_2_paper} and trained on publicly available labeled audio recordings of people reading the New Testament in 1162 languages (more than $45k$ hours). Of the 21 languages in our dataset, this checkpoint is trained\footnote{\url{https://dl.fbaipublicfiles.com/mms/asr/mms1b_all_langs.html}} on Toba Qom (\texttt{tob}) and Greek (\texttt{ell}), of which  Cypriot Greek (\texttt{el-CY}) is a dialect.

XLS-R \citep{xls_r_paper} is also based on the Wav2Vec 2.0 architecture and trained on $436k$ hours of publicly available unlabeled speech recordings. The training covers 128 languages. Of the 21 languages in our dataset, this model is trained on 2 hours of Scots (\texttt{sco}) and $17k$ hours of Greek (\texttt{ell}), of which Cypriot Greek (\texttt{el-CY}) is a dialect.

Whisper is trained on $680k$ hours of weakly labeled data across 99 different languages, collected from the internet \citep{whisper_l_v3_paper} using a sequence-to-sequence transformer architecture. Of the 21 languages in our dataset, this model is trained only on Greek (\texttt{ell}), of which Cypriot Greek (\texttt{el-CY}) is a dialect.


\subsection{Pre-processing}
All models use a common audio and text pre-processing pipeline. Audio files are loaded from disk with soundfile and converted to $16$ kHz mono \verb|float32| waveforms. Entries with non-positive duration or missing/empty transcriptions are removed before further processing. Transcripts are normalized using a light-weight but language-agnostic text cleaning function. After cleaning, any remaining empty or whitespace-only transcripts are discarded.
To control pathological outliers in utterance length, we compute per-language duration statistics using a separate corpus analysis script. During training, we drop any train/validation utterances with raw duration above the $97.5$th-percentile duration (\verb|p97_5_sec|) per language to avoid running into CUDA errors during training.

For CTC-based MMS and XLS-R models, we construct a character-level vocabulary per language by aggregating all unique characters from the cleaned train and validation transcripts. Space is replaced by a word-delimiter symbol ($|$), and special \verb|[UNK]| and \verb|[PAD]| tokens are appended. For Whisper, we reuse the pre-trained tokenizer without modification and rely on its internal normalization and special tokens.


\subsection{Fine-tuning Strategies}
For MMS and XLS-R we adopt a parameter-efficient fine-tuning regime based on adapter layers. \citet{peft_paper} proposed adapter modules as a means to introduce trainable layers in existing architectures to allow parameter-efficient fine-tuning. In MMS, we follow the adapter design described in the HuggingFace blog by \citet{mms_hf_aft_blog}: the base encoder is loaded, lightweight bottleneck adapters are initialized in each encoder block, and the underlying pre-trained parameters are frozen. Only the adapters and the language-specific CTC head are updated during fine-tuning, resulting in a small fraction ($~0.25\%$) of trainable parameters relative to the $1$B-parameter backbone while preserving its cross-lingual representations.

For XLS-R, we mimic this approach by enabling attention adapters and follow the settings given by \citet{xlsr_hf_aft_blog}.

Whisper Large-v3 is fine-tuned using full-model sequence-to-sequence training as explained by \citet{whisper_hf_aft_blog}. In all Whisper experiments, we keep the pre-trained tokenizer and text normalization behavior intact, following the original Whisper training and evaluation protocol \citep{whisper_l_v3_paper}.


\subsection{Training Details}
Training is implemented using the Hugging Face \verb|Trainer| (for MMS and XLS-R) and \verb|Seq2SeqTrainer| (for Whisper) APIs with language-specific adapters and tokenizers. For MMS and XLS-R, we train for $15$ epochs by default, with a per-device batch size of $2$ and gradient accumulation over $8$ steps (effective batch size of $16$ utterances). We use AdamW with a learning rate of $1 \times 10^{-3}$, $100$ warmup steps, and gradient checkpointing enabled to reduce memory footprint.

For Whisper, we fine-tune  \verb|WhisperForConditionalGeneration| using $8$ epochs, the same nominal batch size and gradient accumulation schedule, and a learning rate of $1 \times 10^{-5}$. We started our experiments with an initial set of hyperparameters inspired by Hugging Face blogs \citep{mms_hf_aft_blog, xlsr_hf_aft_blog, whisper_hf_aft_blog} and then iteratively tuned them based on the initial results and training logs.

For all models, we save checkpoints every $100$ steps, evaluate every $100$ steps, retain at most $4$ checkpoints per run, and load the best checkpoint according to validation Word Error Rate (\texttt{WER}).

\subsection{Language Model Decoding}
For the CTC-based MMS and XLS-R models, we optionally augment greedy decoding with external n-gram language models using \verb|pyctcdecode|\footnote{\url{https://github.com/kensho-technologies/pyctcdecode}}, but without an explicit n‑gram language model. After training a model for a given language, we collect the cleaned training transcripts and derive a unigram word list, which provides a lexicon and approximate word frequencies. We then construct the CTC label set from the tokenizer vocabulary in index order, mapping the pad token to the CTC blank (empty string) and the word-delimiter token to a space, following standard practice for CTC decoding.
In this configuration, pyctcdecode performs beam search using the CTC scores and the unigram vocabulary, but does not incorporate any KenLM n‑gram probabilities \citep{heafield-2011-kenlm}. That is, decoding corresponds to ``beam search with unigrams (no ARPA)'' in our logs. We apply this unigram-only beam search decoder only at evaluation time on the validation and test splits; all training and model selection are based on greedy CTC decoding. For Whisper, we do not use any external language model. 


\subsection{Evaluation Metrics}
We report word error rate (\texttt{WER}) as the primary evaluation metric and character error rate (\texttt{CER}) as a secondary metric, computed using the \verb|WER| and \verb|cer| implementations from the Hugging Face \verb|evaluate| library. For CTC-based MMS and XLS-R models, we convert logits to token sequences via argmax (greedy decoding) or beam search (with or without LM), then replace any $-100$ labels with the tokenizer’s pad token id before decoding. For Whisper, the predictions are generated token sequences; we similarly replace masked label positions with the pad token before decoding. 

\subsection{Compute}
All experiments were run on the University of Washington Hyak cluster using single-GPU jobs. All adapter runs used an NVIDIA \verb|L40| or \verb|L40S| GPU with \verb|bf16| or \verb|fp16| training enabled, requiring approximately 1–2 GPU hours per language and split. Whisper fine-tuning, which trains the full encoder–decoder in \verb|float32|, required 2-3 GPU hours per language.