\section{Linguistic Analysis}
\label{sec:lg}

In addition to our \texttt{WER} and \texttt{CER} results, we were interested in analyzing the errors of two languages in particular to determine if there were linguistic features that proved to be especially difficult for the model, or if there were noticeable differences in the kinds of errors made by the model for the two languages. For this closer analysis we chose Scots and Nubi because both are closely associated with a high-resource language and might therefore be expected to perform well, but had different outcomes with the ASR models. Through our analysis we hoped to determine a likely cause for these contrasting results. To perform this error analysis, we used the XLS-R outputs of Scots and Nubi, taking approximately 10\% of the data outputs and performing an error analysis of substitutions, deletions, and insertions. Because Scots shares so much of its lexicon and syntax with English, we were also able to make some phonological inferences for Scots. 

\subsection{Scots}


\input{tables/table_ngram_small}


Of the 75 generated transcripts in Scots, eight were examined. Many of the errors in the Scots data appear to be misalignments with Scots phonology and English orthography. This is particularly apparent in the vowel substitution errors, which appear to align well with Scots vowels. For example, the transcription of “walking” as “walken” is potentially the result of the realized vowel being lower than the “standard” English vowel that is represented by the letter “i.” This word is also an example of a complication that arises when analyzing transcription errors in an orthography that uses digraphs. The use of “ng” to represent the phoneme /\textipa{N}/ turns one phoneme into two characters, and when the model transcribes “walken” with only an “n” instead of an “ng,” it is analyzed as a deletion error, when it is perhaps more linguistically accurate to analyze it as a substitution error between the phonemes /n/ and /\textipa{N}/.

There were several consonant substitutions, particularly in English words, that do not align with a potential phonological and orthographic disagreement between Scots and English. For example, the word “gymnastics” was transcribed as “Jimnastic” and “physique” was transcribed as “fhasic.” This does not appear to represent some underlying phonological difference in the consonants, but rather a failure of the model to select the correct characters. 

It was common for word-final characters to be deleted, both consonants and vowels, as evidenced by words like “hole” becoming “hol” and “chill” becoming “chil.” These characters do not directly represent phones in the spoken data, and as such we would not expect these deletions to be the result of some difference between the Scots phonology and the English orthography. 

As previously noted, much of the Scots lexicon is shared with English. In this dataset, the Scots words that are not common with English tend to be short, high-frequency words. While there were some errors, these uniquely Scots words like \textit{oot} “out” and \textit{maist }“most” tended to be transcribed faithfully. This may be due to their frequency in the data, but it may also be a result of their relatively phonetic spellings, in contrast to the English words.

It’s also important to note the difference in punctuation between the languages’ transcripts. The Nubi data has no punctuation of any kind and relatively little capitalization. In contrast, the Scots transcripts have significant punctuation, including hyphens denoting interrupted speech and apostrophes in contractions. This punctuation might have increased the error rates in Scots, as punctuation increases the number of possible characters for the model to choose from, while not corresponding to phonetic information in the data, and still contributing to error counts. 

\subsection{Nubi}

Our analysis of Nubi was limited by our relative unfamiliarity with the language and orthography of the dataset, meaning phonological and morphosyntactic analysis was not possible. However, we were able to identify some potential trends in the errors. Of the 100 transcripts in the test, ten were analyzed for common substitution, deletion, and insertion errors in comparison with the provided gold transcriptions. 

Certain substitutions between vowels appeared throughout the entries, especially between the pairs ``i'' and “e” and “o” and “u”. Assuming that the orthography used in this dataset is typical, this is likely representative of the relative similarities of the pairs of front unrounded vowels and back rounded vowels. Similarly, the nasals “m” and “n” were commonly substituted, which is unsurprising given their similarity and the generally lower perceptual differences between nasals as compared to vowels or non-nasal consonants \cite{hura1992role}. There was also a substitution that occurred in a specific environment, which may indicate an error happening with the LM. In all instances of the word “ab”, the model transcribed “al”, but the “b” > “l” substitution did not occur outside of that specific environment. 

Vowel deletions were very common throughout the data, for each of the five vowels, words initially, medially, and finally. There does not appear to be a significant pattern to the single vowel deletions. There was a pattern, however, with the deletion of repeated vowels, which may indicate a problem with the model. Every time there was a repeated vowel in the gold transcript, the ASR model would transcribe only the first of the repeating vowels and delete the rest. Consonant deletions were less common than vowel deletions, generally, and the majority of the deleted consonants were nasals. In addition to the nasals, there was a trend of ``w'' being deleted intervocalically. Without knowledge of the orthography, it is difficult to confidently infer a reason for this, but Nubi does have the phoneme /w/, which may be represented by the letter ``w'' \cite{wellens2003arabic}. If so, this could indicate a failure of the model to recognize the approximant in this environment, but it may also be a reflection of a phonological phenomenon. 

In two of the transcripts, there were English loanwords in the transcript that proved to be difficult for the model to transcribe. The loanwords included ``mindset'', ``school'', ``government'', ``busy'', ``creative'', and ``typhoid''. Some of these words resulted in a series of substitution errors, for example, “mindset” became “ma endist” and “busy” became “bse,” whereas “school” was deleted in its entirety, and “creative” resulted in a mix of substitution and deletion with “kwet.” This indicates that the model struggles with multilingual ASR.

The most surprising and egregious errors made by the model were insertions that significantly outnumbered the actual word count of the gold transcript. For example, one entry was eight words long, but the model returned a transcript that was 120 words long; another was five words long and returned a transcript that was 72 words long. This would significantly affect \texttt{WER} and may be the cause of Nubi's relatively poor results in the tests. 

\subsection{Scots vs. Nubi: A Comparative View}

Both languages stand in a ``shadow'' relationship to a high-resource language — English for Scots and Arabic for Nubi - with varied error patterns. Scots errors cluster at the interface between Scots phonology and English orthography: vowel substitutions reflecting genuine Scots realizations rendered in English-like spellings, non-phonetic word-final deletions, and punctuation-driven noise. The model carries a strong English prior, producing Scots-adjacent transcripts mis-anchored to English conventions. Nubi errors, by contrast, lack any comparable anchor: segmental confusions are broader and less systematic, and the most consequential failures are catastrophic insertion blowups (e.g., 8-word references producing 120-word hypotheses), consistent with a decoder that fails to terminate reliably. Proximity to a high-resource language thus appears double-edged — it supplies useful priors but may also impose a wrong frame, whereas its absence can leave the decoder structurally unstable.