\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{spverbatim}
\usepackage{tabularx}
\usepackage{amsmath}

\usepackage{cleveref}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Stylistic Transfer from Annotator Communities to Large Language Models}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Jay Chooi \\
  Harvard University\\
  \texttt{jeqin\_chooi@college.harvard.edu}\\}

%\author{
%  \textbf{First Author\textsuperscript{1}},
%  \textbf{Second Author\textsuperscript{1,2}},
%  \textbf{Third T. Author\textsuperscript{1}},
%  \textbf{Fourth Author\textsuperscript{1}},
%\\
%  \textbf{Fifth Author\textsuperscript{1,2}},
%  \textbf{Sixth Author\textsuperscript{1}},
%  \textbf{Seventh Author\textsuperscript{1}},
%  \textbf{Eighth Author \textsuperscript{1,2,3,4}},
%\\
%  \textbf{Ninth Author\textsuperscript{1}},
%  \textbf{Tenth Author\textsuperscript{1}},
%  \textbf{Eleventh E. Author\textsuperscript{1,2,3,4,5}},
%  \textbf{Twelfth Author\textsuperscript{1}},
%\\
%  \textbf{Thirteenth Author\textsuperscript{3}},
%  \textbf{Fourteenth F. Author\textsuperscript{2,4}},
%  \textbf{Fifteenth Author\textsuperscript{1}},
%  \textbf{Sixteenth Author\textsuperscript{1}},
%\\
%  \textbf{Seventeenth S. Author\textsuperscript{4,5}},
%  \textbf{Eighteenth Author\textsuperscript{3,4}},
%  \textbf{Nineteenth N. Author\textsuperscript{2,5}},
%  \textbf{Twentieth Author\textsuperscript{1}}
%\\
%\\
%  \textsuperscript{1}Affiliation 1,
%  \textsuperscript{2}Affiliation 2,
%  \textsuperscript{3}Affiliation 3,
%  \textsuperscript{4}Affiliation 4,
%  \textsuperscript{5}Affiliation 5
%\\
%  \small{
%    \textbf{Correspondence:} \href{mailto:email@domain}{email@domain}
%  }
%}

\begin{document}
\maketitle


\begin{figure*}[!htb]
    \centering
    \includegraphics[width=\linewidth]{img/sighum.png}
    \caption{Our setup. (First row) Two similar model families with base and post-trained variants, with Nigerian annotators present in one but not in the other. (Second row) All four models are prompted on AlpacaEval and MT-Bench to generate model-specific corpora. (Third row) The similarity between the model corpora and the corpora in Nigerian English and US English is computed using the Jensen-Shannon distance (JSD). (Fourth row) The difference in JSD is calculated for each model. (Last row) The difference in JSD differences between the base and post-trained variant is visualized.}
    \label{fig:overview}
\end{figure*}



\begin{abstract}
Large language models (LLMs) are post-trained on human feedback collected from annotator communities, yet the linguistic influence of these annotator communities on language models remains poorly understood. We investigated the stylistic transfer from Nigerian annotators to the LLaMA family of models through a natural experiment with LLaMA 2 and LLaMA 3.1, as their release dates are separated by the shutdown of a major data annotation service provider in Nigeria. We generated corpora from both model families and measured linguistic style by computing the difference-in-difference of the Jensen-Shannon distance on the bigram distribution between model outputs and corpora of Nigerian English and US English. We found that, although both pre-trained model variants exhibit similar proximity to both English variants, the LLaMA 2 post-trained model moved toward Nigerian English, while the LLaMA 3.1 post-trained model moved away from Nigerian English. Qualitatively, we found that post-trained LLaMA 2 models used significantly fewer contractions, in line with Nigerian English speakers opting to use a formal register due to its role as an index of knowledgeability. Our findings suggest that annotator communities can imprint linguistic style on large language models, with potential implications such as a disproportionately higher false positive rate in AI plagiarism detection for users who share a linguistic style with annotator communities.
\end{abstract}


\section{Introduction}
\label{sec:introduction}

Large language models (LLMs) undergo two phases of training. The first phase, known as pre-training, trains the model on a large text corpus to acquire language fluency and world knowledge \cite{radford_improving_2024}. Models that underwent the first phase are known as \emph{base} models. The second phase, known as post-training, includes multiple stages. The first stage is usually supervised fine-tuning (SFT), where the model is trained on a dataset to act like a helpful assistant. The second stage is usually reinforcement learning from human feedback \cite{ouyang_training_2022}, where humans provide direct feedback to the model. Specifically, a human \emph{annotator}, often paid, will choose between various model outputs given the same instruction, or prompt. The language model will then be shaped toward producing outputs that are preferred by the annotator. Models that underwent the post-training phase are known as \emph{chat} or \emph{instruct} models.

LLM companies hire annotators to train their models to be useful for humans. However, beyond transferring human preference for assistant-like behavior, it is not known whether the \emph{linguistic style} of the annotator is also transferred to the post-trained model. Annotator sourcing is often done in countries where labor costs are low \citep{perrigo2023openai}, as annotation work could be done remotely under the assumption that the ideal model preferred by these annotators is similar to the ideal model preferred by the user base. Countries with known annotator communities include Nigeria, Kenya, and the Philippines \cite{wang_whose_2022}. This sociolinguistic imbalance of employing annotator communities that speak a different variant of English could transfer subtle stylistic biases to the LLM. Prior work has noted a difference in the statistical distribution of machine-generated text and human text (see Section \ref{sec:related-work}), but none has empirically traced this difference to annotator communities. 

In this paper, we study the stylistic transfer from Nigerian annotator communities to LLaMA models. We compared outputs from LLaMA 2 models \cite{touvron_LLaMA_2023} and LLaMA 3.1 models \cite{grattafiori_LLaMA_2024}, the latter of which were released after a major data annotation service provider in Nigeria was shut down. We compute bigram frequencies and calculate the distance in distribution of model outputs to established corpora of Nigerian English and US English using the Jensen-Shannon distance (JSD). We found that although both LLaMA 2 and LLaMA 3.1 models have similar JSD for their base models, the post-trained LLaMA 2 model shows closer proximity to Nigerian English than the post-trained LLaMA 3.1 model which moved away from Nigerian English. This finding suggests that the LLaMA 2 post-trained model could have inherited the linguistic style of Nigerian English through its Nigerian annotators. Furthermore, we extracted qualitative examples that align with current literature on the differences between Nigerian and US English. In Section \ref{sec:qual-var}, we show that post-trained LLaMA 2 avoided the use of contractions (it's, don't), which is the case with Nigerian English speakers as well.

Our findings suggest that annotator communities form an unmentioned but measurable influence on the linguistic drift of large language models. As comparison with machine-generated text distribution is often used as a basis for plagiarism or dishonest AI-use detectors \cite{mitchell_detectgpt_2023}, we note that such plagiarism detectors could be disproportionately biased against Global South communities who have annotators in the post-training pipeline.

\section{Related Work}
\label{sec:related-work}

\paragraph{Sociolinguistics of LLMs} \citet{grieve_sociolinguistic_2025} argued that the current misalignment of large language models stems from selective procurement of training data and annotators that are easily obtainable but not necessarily representative of the user base of the model. Our paper provides evidence toward the influence of annotators by tracing the annotator community's linguistic style through the post-training pipeline.

\paragraph{Bias from the training process} There is a rich literature \citep{hovy_social_2016,sap_risk_2019,https://doi.org/10.1111/lnc3.12432,casper_open_2023} on how skewed dataset curation and unrepresentative annotator selection can bias NLP systems. Our paper demonstrates that aside from explicit preferences, the linguistic style of the LLM itself could be influenced by annotators.

\paragraph{World English and LLM English} \citet{juzek_why_2024} explored the transformation of Scientific English after the introduction of large language models by identifying ``focal words'' that exploded in popularity in published abstracts and coincided with frequent model-generated completions. A follow-up work found that RLHF does affect word choice of the model \citep{juzek_word_2025}. We expand beyond word choice to analyze linguistic styles via distributional shifts of bigrams. Furthermore, \citet{liang_gpt_2023} found that AI plagiarism detectors are biased against non-native speakers. Here, we draw the connection that the detection bias could stem from AI-generated English being closer to that of annotator communities due to their involvement in the training process.

% \paragraph{n-grams} \cite{lu_ai_2025} used \(n\)-grams to characterize creativity and for language model output detection. We used bigrams in our methodology to study the distributional shift of LLM English to Nigerian and US English.


\section{Methods}
\label{sec:methods}

The code for the methods described here is released online. \footnote{Code is available at \url{https://github.com/jeqcho/LLaMA-alpacaeval-mt-bench} and \url{https://github.com/jeqcho/tracing-gpt-annotators} for corpora production and analysis, respectively.}

% The code for the methods described here is released online. \footnote{Code is available at REDACTED and REDACTED for corpora production and analysis, respectively.}

\subsection{Corpora Identification and Production}
We first identify an annotator community that is hypothesized to have strong linguistic influence on the language model. We chose the Nigerian annotator community as it has been hypothesized that the recent explosion in the popularity of the word ``delve'' in written material is directly linked to the popular use of ``delve'' in Nigerian English \cite{hern_techscape_2024}.

We then identify two model families where one was likely exposed to the Nigerian annotator community while the other likely did not. To control for stylistic differences that might stem from cumulative algorithmic improvements over time or pre-training material that differs across companies, we sought to find model families from the same company and released close in time. The two model families identified are Meta's LLaMA 2 family \cite{touvron_LLaMA_2023} and LLaMA 3.1 family \cite{grattafiori_LLaMA_2024}. Between the release dates of these two families, Scale AI, the data annotator company widely used by LLM companies, shut off access to its annotation platform Remotasks in Nigeria (see Table \ref{tab:LLaMA-remotasks} for a timeline). It is highly likely that Meta used Scale AI's services to post-train their AI models, with Meta later buying 49\% of Scale AI \cite{godoy_metas_2025}. We skipped LLaMA 3 because it is plausible that post-training of LLaMA 3 could still have involved Nigerian annotators as its release date is close to when Remotasks cut off access. The models we chose are \verb|LLaMA-2-13b-hf|, \verb|LLaMA-2-13b-chat-hf|, \verb|LLaMA-3.1-8B|, \verb|LLaMA-3.1-8B-Instruct|. In the results (see Figure \ref{fig:jsd_diff_in_diff_plot}), we verified that the difference in pre-training between these families (if any) did not have any statistical influence on the style of English used, while there is a large difference in style after post-training.

We now turn to obtaining a distribution of the model English, US English, and Nigerian English. For the model English, we did not find any pre-existing corpus online that is fully generated by LLaMA 2. Therefore, we generated our own corpus. We ran both LLaMA 2 and LLaMA 3.1 models on AlpacaEval \cite{li_alpacaeval_2023} and MT-Bench \cite{zheng_judging_2023}. Examples of prompts from these datasets and the model outputs can be found in Appendix \ref{sec:alpacaeval-mtbench}. Running the generations took around 2 hours of compute time on a \verb|g6e.xlarge| AWS EC2 instance. We set max new tokens to be 512, temperature as 0.7, and top-p as 0.9. For US English, we obtained a sample with 8.9 million words from the Corpus of Contemporary American English (COCA) \cite{davies_corpus_2008}. For Nigerian English, we used the written component (400k words) of the International Component of English - Nigeria \cite{gut_international_2014}. More details and instructions on how we downloaded these datasets can be found in Appendix \ref{sec:corpora}.

\subsection{Quantitative Detection of Linguistic Style Transfer}

We measure the closeness of the LLM output distribution to US English and Nigerian English. We apply the Jensen-Shannon distance (JSD), which is the square root of the Jensen-Shannon divergence \cite{lin_divergence_1991}. Two disjoint distributions have a JSD of 1, while a distribution has a JSD of 0 with itself. The JSD is a metric, and hence it is symmetric. The JSD ranges from 0 to 1. We construct the distribution through the frequency distribution of bigrams after lowercasing all corpora.

\newcommand{\JSD}{\text{JSD}}
Since we are interested in the effect of post-training conditional on the presence of Nigerian annotators, we adopt a difference-in-difference approach. Specifically, we first calculate the JSD between each model distribution and each English variant. For each model variant, we then calculate the difference in JSD from the Nigerian and US variants. We then compute the difference of this difference across base and post-trained model variants. The \(\Delta \JSD\) y-axis title in Figure \ref{fig:jsd_diff_in_diff_plot} refers to \(\JSD(\text{model English},\text{US English})-\JSD(\text{model English},\text{Nigerian English})\). We are interested in \(\Delta \JSD_\text{post-trained}-\Delta \JSD_\text{base}\) for both LLaMA 2 and LLaMA 3.1 models. If \(\Delta \JSD_\text{post-trained}-\Delta \JSD_\text{base}\) is positive, then the post-trained model is closer to Nigerian English than the base model, and vice versa.

\subsection{Hunting for Qualitative Variables}

Here we describe a method to surface the bigrams that are especially pronounced in both the Nigerian corpus (when compared to the US corpus) and the post-trained LLaMA 2 corpus (as compared to the pre-trained version, and LLaMA 3).

\begin{table}[h!]
\centering
\begin{tabular}{ll}
\hline
\textbf{Subscript} & \textbf{Description} \\
\hline
$\mathrm{L2\text{-}chat}$ & LLaMA-2 Chat (post-trained) \\
$\mathrm{L2\text{-}base}$ & LLaMA-2 Base (pre-trained) \\
$\mathrm{L3\text{-}chat}$ & LLaMA-3.1 Chat (post-trained) \\
$\mathrm{L3\text{-}base}$ & LLaMA-3.1 Base (pre-trained) \\
$\mathrm{NG}$ & Nigerian English corpus \\
$\mathrm{US}$ & US English corpus \\
\hline
\end{tabular}
\caption{Subscripts used in bigram frequency notation}
\label{tab:sub-corpus}
\end{table}

Using the subscript notation introduced in Table \ref{tab:sub-corpus}, let the set of corpora be
\[
\begin{aligned}
C = \{ & \mathrm{L2\text{-}chat},\ \mathrm{L2\text{-}base}, \\ 
       & \mathrm{L3\text{-}chat},\ \mathrm{L3\text{-}base},\ \mathrm{NG},\ \mathrm{US}\}.
\end{aligned}
\]

Let \(B\) be the set of all possible English bigrams. For any bigram \(i\in B\), denote the frequency of that bigram in a corpus \(c\in C\) as \(f_c(i)\). We compute relative bigram frequencies as usual:
\[
r_c(i) = \frac{f_c(i)}{ \sum_{j\in B}f_c(j) }.
\]

We compute the Nigerian score of a bigram \(i\in B\) as follows
\[
\mathrm{score}_\mathrm{NG}(i)=r_\mathrm{NG}(i)-r_\mathrm{US}(i)
\]

We compute the post-trained scores of a bigram \(i\in B\) as follows
\begin{align*}
s_\mathrm{L2}(i)=r_\mathrm{L2\text{-}chat}(i)-r_\mathrm{L2\text{-}base}(i)\\
s_\mathrm{L3}(i)=r_\mathrm{L3\text{-}chat}(i)-r_\mathrm{L3\text{-}base}(i)
\end{align*}

We compute the relative score of a bigram \(i\in B\) toward post-trained LLaMA 2 as follows
\begin{align*}
    \mathrm{score}_{\mathrm{L2\text{-}chat}}(i)=s_{\mathrm{L2}}(i)-s_{\mathrm{L3}}(i)
\end{align*}

Finally, we compute the \emph{divergence score} of each bigram \(i\in B\) as follows
\begin{align*}
    \mathrm{divergence}(i)=\mathrm{score}_\mathrm{NG}(i)\times\mathrm{score}_{\mathrm{L2\text{-}chat}}(i)
\end{align*}

A bigram that achieves a high positive divergence score could be explained by appearing much more often in the Nigerian corpus (relative to the US corpus) \emph{and} appearing much more often in the post-trained LLaMA 2 corpus (relative to its pre-trained version) \emph{and} this difference in the difference in frequency in the LLaMA 2 corpus is much more than that observed in the LLaMA 3 corpus. Similarly, a high positive score is also achieved when it appears more often in the US corpus relatively, and in the post-trained LLaMA 3 corpus relatively. To tell apart these two causes, we can examine the sign of \(\mathrm{score}_\mathrm{NG}(i)\), as a positive sign implies the former, and a negative sign the latter.


\section{Results}
\label{sec:results}

\subsection{Distributional Results}
The differences in JSD are plotted in Figure \ref{fig:jsd_diff_in_diff_plot}. The exact JSD numbers are available in Table \ref{tab:jsd_corpora}. As noted in Section \ref{sec:methods}, the base model difference in JSD for both LLaMA 2 and LLaMA 3.1 is similar and suggests that the pre-training pipeline for both model families has no significant impact on style, likely due to similar pre-training curriculum. For post-training, however, the LLaMA 2 Chat model is closer to Nigerian English relative to its base model variant, while the LLaMA 3.1 Instruct model moved further away from Nigerian English relative to its base model variant. Since LLaMA 2 was released before the shutdown of Remotasks in Nigeria, while LLaMA 3.1 was released months after, it is plausible that this difference in post-training JSD is attributable to LLaMA 2 having part of its data annotation work done by Nigerian workers while LLaMA 3.1 did not.


\begin{figure}[!htb]
    \centering
    \includegraphics[width=\linewidth]{img/jsd_diff_in_diff_plot.png}
    \caption{The base models of LLaMA 2 and LLaMA 3 show similar differences in the Jensen-Shannon distance (JSD) between Nigerian and US English, indicating similar styles along the Nigerian -- US English continuum. After post-training, LLaMA 2 showed an increase in the difference of JSD (moving closer to Nigerian English), while LLaMA 3 showed a decrease that is more muted in magnitude in the difference of JSD (moving away from Nigerian English).}
    \label{fig:jsd_diff_in_diff_plot}
\end{figure}

\subsection{Qualitative Results}
\label{sec:qual-var}

We report the top 10 bigrams by divergence score in Table \ref{tab:divergence}, conditioned on the sign of \(\mathrm{score}_{\mathrm{NG}}\).

\begin{table}[h!]
\centering
\begin{tabular}{cll}
\hline
\textbf{Rank} &   \textbf{L2-NG} &\textbf{L3-US}\\
\hline
1  & of the   & do you \\
2  & it is    & it's a \\
3  & in the   & and a \\
4  & and the  & i was \\
5  & should be & to a \\
6  & that the & and then \\
7  & is not   & it's not \\
8  & can be   & trying to \\
9  & such as  & like a \\
10 & by the   & more than \\
\hline
\end{tabular}
\caption{Top 10 bigrams by $\mathrm{divergence}$ score, split by positive $\mathrm{score}_{\mathrm{NG}}$ (L2–NG) vs. negative $\mathrm{score}_{\mathrm{NG}}$ (L3–US). The L2-NG column shows the bigrams that are over-represented in both the Nigerian corpus and the post-trained LLaMA 2 corpus. The L3-US column shows the bigrams that are over-represented in both the US corpus and the post-trained LLaMA 3 corpus.}
\label{tab:divergence}
\end{table}

One major result is that contractions appeared on the L3-US list (``it's a'', ``it's not'') while no contractions appeared on the L2-NG list where instead bigrams that could be contracted appeared, like ``it is'' and ``is not''. This aligns with existing literature where American English generally allows for the use of an informal register that involves contractions (it's, don't), while Nigerian English tends to avoid the informal register and instead predominantly uses the formal register. In fact, Nigerians use the phrase ``colloquial English'' to describe bad English \cite{kperogi_divided_2010}. Furthermore, for most Nigerians, English is exclusively taught in school instead of as a native language used within families, hence formality is especially emphasized through the schooling process. \citet{christiana-oluremi_lexico-grammatical_2013} further describes that the use of the formal register of English in Nigeria serves as an index of knowledgeability and learnedness. Having the use of a formal register as an index of educatedness may have prompted the consistent use of formal English by Nigerian English speakers more often than American speakers.

One potential counterargument is that the younger generation of Nigerians, who might constitute the bulk of the Nigerian annotators, does use contractions. Indeed, \citet{ebibi_study_2018} has found that 87\% of students of a local Nigerian college used contractions in their writing. However, the administration of such a survey implies that there is still strong stigma among English speakers in Nigeria to use contractions in their writing, with the paper further describing that most students who use contractions claimed that using contractions affected their fluency of English.

Another counterargument is that the lack of contractions in L2-NG might simply be a generic post-training artifact toward formality. However, if formality were a general post-training tendency, we would expect both LLaMA 2 and LLaMA 3.1 to avoid contractions after post-training. Instead, LLaMA 3.1 moves toward contractions while LLaMA 2 moves away, suggesting that the contraction avoidance is specific to LLaMA 2's post-training pipeline. This is consistent with the presence of Nigerian annotators in post-training LLaMA 2 but not LLaMA 3.1, aligning with the fact that Nigerian annotators prefer formality and avoid contractions.

\subsubsection{To delve or not to delve?}\label{ssc:to-delve-or-not-delve}

To move toward a resolution of whether the increase of the usage of the word ``delve'' in modern online text is because modern LLMs are trained by Nigerian annotators, we report in Table \ref{tab:delve} the prevalence of bigrams that involve the substring ``delve''.

\begin{table}[h!]
\centering
\begin{tabular}{lrr}
\hline
\textbf{Corpus} & \textbf{\% of delve bigrams} & \textbf{``delve'' / all} \\
\hline
NG       & 0.001247 & 5 / 400{,}782 \\
US       & 0.000314 & 22 / 7{,}004{,}385 \\
L2-base   & 0.000390 & 1 / 256{,}597 \\
L2-chat   & 0.003131 & 7 / 223{,}695 \\
L3-base   & 0.000821 & 2 / 243{,}634 \\
L3-chat   & 0.004925 & 12 / 243{,}718 \\
\hline
\end{tabular}
\caption{Frequency of bigrams involving the substring ``delve''. }
\label{tab:delve}
\end{table}
Directionally, the Nigerian corpus uses ``delve'' roughly 4x as much as the US corpus, aligning with the common belief that Nigerians use the phrase more often than American speakers \cite{hern_techscape_2024}. Meanwhile, the post-trained LLaMA 2 model uses ``delve'' 8x more often than its pre-trained version, while the LLaMA 3.1 model only uses it 6x as often. Since the post-trained LLaMA 2 sees a larger increase in ``delve'' usage than LLaMA 3.1, it is tempting to declare that indeed Nigerian annotators have made LLaMA 2 delve into the usage of more ``delves'' compared to LLaMA 3.1, but the rare occurrences of such bigrams reported in Table \ref{tab:delve} render us unable to draw such strong conclusions. We leave the continued investigation of the recent prominence of the word ``delve'' to future work.

\section{Discussion}

There is no Standard English \cite{lippi-green_english_2012}. Nigerian English and US English are spoken by English speakers in Nigeria and the US respectively, and have unique features that distinguish the two. After pre-training on the Internet, which is predominantly in US English, and post-training through Nigerian English speakers, the resulting variant of English used by \verb|LLaMA-2-13b-chat-hf| may be thought of as a variant of English in its own right.

This difference between annotator population and user population can be thought of as contributing to the formation of a \emph{community of practice} \cite{eckert_communities_1992} among large language models, where participation is defined by post-training with Nigerian English speakers. \verb|LLaMA-2-13b-chat-hf| model will then be a useful case study of this community of practice, as well as other models pre-trained with Internet text but post-trained with Nigerian English speakers. Under the lens of the social meaning of linguistic style \cite{podesva_three_2008}, the JSD difference could serve as a linguistic dimension for researchers to contrast the linguistic styles of different post-trained models, with Nigerian post-training involvement serving as its indexicality or social meaning.

\section{Limitations}

One key limitation is that the US English variant corpus might be different from the Nigerian English variant corpus not merely in style but also in domain. It is possible that the explanatory power behind the domain difference in the corpora might explain the difference-in-differences of JSD more than the stylistic differences in the corpora. This possible mechanism could be accounted for if we have two corpora similar in domain but one in US English while the other is in Nigerian English. We do not find any such corpora available online.

Another limitation is that there might be other differences in the post-training pipeline of LLaMA 2 and LLaMA 3.1 aside from differences in annotator communities, namely, that of curation of instruction-tuning datasets for supervised fine-tuning (SFT). Future large-scale experiments where identical base models with the same SFT treatment are post-trained with annotator communities from distinct English-speaking populations would more cleanly isolate this effect.


\section{Further Directions}

The subtle statistical difference in language use between LLM outputs and user outputs in the developed world is often used to flag dishonest AI use \cite{wu_survey_2025}. As our results suggest that the distribution of model outputs move toward English varieties that are closer to annotator communities in the Global South, one concern could be that such LLM detectors could disproportionately misclassify genuine human outputs in the Global South as machine output. Further research is welcome in this area, including running such LLM detectors on corpora of various English varieties to see if there is a consistent higher false positive rate on English varieties associated with annotator communities.

As investigated in \Cref{ssc:to-delve-or-not-delve}, we couldn't establish strong evidence that Nigerian annotators are causally responsible for the recent increase in the usage of the word ``delve'' in online text. We leave it to future work to establish or fully disprove the causal link between Nigerian annotators and LLMs' prevalent usage of ``delve''.

Replicating the experiment using LLaMA 2 7B instead of LLaMA 2 13B might be a fairer comparison to LLaMA 3.1 8B due to their closer parameter count.

More variants of annotator and non-annotator English variants could be helpful to determine the strength and stability of this effect. One data-rich non-annotator English variant would be the New Zealand English variant \cite{victoria_university_of_wellington_school_of_linguistics__applied_language_studies__university_of_zurich_ice_project_ice-new_1999}, since it is likely that the number of New Zealanders in the post-training annotator community is much less compared to that of Nigeria due to their relative population size and the popularity of data annotation work in Nigeria as the labor cost is cheaper. Other annotator communities that have corpora online include Kenya \cite{ice_project_university_of_zurich__chemnitz_university_of_technology__university_of_bayreuth_ice-east_1999} and the Philippines \cite{ice_project_university_of_zurich__de_la_salle_university_manila_ice-philippines_2004}. Since the Remotasks shutdown also affected Kenyan annotators but not Philippines annotators, a direct future work would be to analyze if a similar drift occurred for Kenyan English but not Philippine English.

\section*{Acknowledgements}
The author thanks Professor Kathryn Franich and Alice Wang for their guidance. The author also thanks the anonymous reviewers for their helpful feedback.



\bibliography{custom,references}

\appendix


\section{Timeline of LLaMA releases and Remotasks Nigeria shutdown}
\label{sec:timeline}

See Table \ref{tab:LLaMA-remotasks}. We skipped LLaMA 3 from our analysis as its release is only one month after the closure of Remotasks in Nigeria. We are more confident that the post-training of LLaMA 3.1 does not involve Nigerian annotators.

\begin{table*}[!htbp]
\centering
\caption{Timeline of LLaMA releases and Remotasks Nigeria shutdown}
\begin{tabularx}{\textwidth}{llX}
\hline
\textbf{Date} & \textbf{Event} & \textbf{Notes} \\
\hline
Jul 18 2023 & LLaMA 2 release & Meta AI releases LLaMA 2 models (7B–70B) \cite{touvron_LLaMA_2023} \\
Mar 7 2024 & Remotasks access loss & Nigerian, Kenyan, and Pakistani workers lose access \cite{brandom_scale_2024} \\
Mar 28 2024 & Rest of World report & Confirms large-scale lockouts of annotators \cite{brandom_scale_2024} \\
Apr 18 2024 & LLaMA 3 release & Meta AI launches LLaMA 3 (8B, 70B) \cite{grattafiori_LLaMA_2024} \\
Jul 23 2024 & LLaMA 3.1 release & Meta AI releases LLaMA 3.1 (8B, 70B, 405B) \cite{grattafiori_LLaMA_2024} \\
\hline
\end{tabularx}
\label{tab:LLaMA-remotasks}
\end{table*}


\section{AlpacaEval and MT-Bench Model Outputs}
\label{sec:alpacaeval-mtbench}
\subsection{AlpacaEval Example}

\textbf{Instruction:} What are the names of some famous actors that started their careers on Broadway?

\subsubsection{LLaMA-2-13B Base Output}
\begin{spverbatim}
Bette Davis, Marlon Brando, Judy Garland, and James Dean were all
famous actors that started their careers on Broadway. They were all
in Broadway shows before going on to be movie stars. Judy Garland
played Dorothy in the Wizard of Oz, Bette Davis played Medea,
Marlon Brando played Stanley Kowalski in A Streetcar Named Desire,
and James Dean played Cal Trask in East of Eden.
\end{spverbatim}

\subsubsection{LLaMA-2-13B Chat Output}
\begin{spverbatim}
Many famous actors got their start on Broadway before transitioning
to film and television. Here are some examples:

1. Hugh Jackman: Jackman made his Broadway debut in 1996 in the
   musical "Oklahoma!" and went on to star in other productions
   such as "The Boy from Oz" and "Les Misérables."
2. Anne Hathaway: Hathaway made her Broadway debut in 2002 in the
   play "The Last Night of Ballyhoo" and later starred in "Rodgers
   & Hammerstein's Cinderella" and "Les Misérables."
3. Jake Gyllenhaal: Gyllenhaal made his Broadway debut in 2002 in
   the play "The Rainmaker" and later starred in "If There Is I
   Haven't Found It Yet" and "Constellations."
4. Idina Menzel: Menzel made her Broadway debut in 1995 in the
   musical "Rent" and went on to star in "Wicked" and "If/Then."
5. Kristin Chenoweth: Chenoweth made her Broadway debut in 1997 in
   the musical "Steel Pier" and went on to star in "Wicked" and
   "The Drowsy Chaperone."
\end{spverbatim}

\subsubsection{LLaMA-3-8B Base Output}
\begin{spverbatim}
What are some examples of plays that were later made into movies?
Broadway is a famous street in New York City. It is also the name
of a famous theatre district in New York City. The area is filled
with theatres that host musicals and plays.
Broadway is a famous street in New York City. It is also the name
of a famous theatre district in New York City. The area is filled
with theatres that host musicals and plays.
\end{spverbatim}

\subsubsection{LLaMA-3-8B Instruct Output}
\begin{spverbatim}
Here are some famous actors who got their start on Broadway:

1. Audrey Hepburn: Began her career on Broadway in the 1950s,
   performing in shows like "High Button Shoes" and "Ondine."
2. Julia Roberts: Made her Broadway debut in the 1987 revival of
   "Steel Magnolias" and later starred in "Six Degrees of
   Separation."
3. Neil Patrick Harris: Got his start on Broadway in the 1990s,
   performing in shows like "Rent" and "Hedwig and the Angry Inch."
4. Hugh Jackman: Starred in the 1999 Broadway production of
   "Sunset Boulevard" and later appeared in "Oklahoma!" and "The
   Boy from Oz."
5. Kristin Chenoweth: Began her career on Broadway in the 1990s,
   performing in shows like "Steel Pier" and "You're a Good Man,
   Charlie Brown."
\end{spverbatim}

\subsection{MT-Bench Example}

\textbf{Question:} Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.

\subsubsection{LLaMA-2-13B Base Output}
\begin{spverbatim}
This travel blog post should be engaging and informative, providing
readers with an inside look at what makes Hawaii a unique and
exciting destination. The post should be well-researched and include
interesting facts and details about the destination, as well as
practical information for travelers. Include a mix of text and
images to illustrate the points made in the blog post and make it
visually appealing.
\end{spverbatim}


\subsubsection{LLaMA-2-13B Chat Output}
\begin{spverbatim}
Title: Aloha! Discovering the Heart of Hawaiian Culture and
Must-See Attractions

Introduction:
Hawaii, the tropical paradise in the Pacific, is a destination that
offers a unique blend of stunning natural beauty, rich cultural
heritage, and unparalleled adventure. Recently, I embarked on a
journey to explore the islands of Oahu, Maui, and Kauai, and I was
left with unforgettable memories and a deeper appreciation for the
Hawaiian culture. In this blog post, I'll share my top cultural
experiences and must-see attractions that will inspire you to plan
your own dream trip to Hawaii.

Cultural Experiences:

1. Attend a traditional Hawaiian luau: A luau is a quintessential
   Hawaiian experience that offers a glimpse into the island's rich
   cultural heritage. I attended a luau at the Waikiki Starlight
   Luau on Oahu, which featured a traditional Hawaiian feast, live
   music, and authentic hula dancing.
2. Visit a Hawaiian temple: Hawaii is home to numerous temples, or
   heiaus, that are sacred to the native Hawaiian people. I visited
   the Pu'uhonua o Honaunau National Historical Park on the Big
   Island, which is a historic temple ground.
3. Learn to hula: Hawaiian dance is an essential part of the
   island's culture, and I had the opportunity to learn the art of
   hula from a local dance teacher.
\end{spverbatim}

\subsubsection{LLaMA-3-8B Base Output}
\begin{spverbatim}
Incorporate SEO strategies, formatting, and a conclusion for a
well-rounded piece.
Aloha, fellow travelers! Hawaii is a dream destination, and I'm
here to share my recent trip to the beautiful islands. Join me as
we explore the vibrant culture, stunning landscapes, and
unforgettable experiences that await in this tropical paradise.
Hawaii is a melting pot of cultures, with influences from
Polynesia, Asia, and the United States. As I stepped off the plane,
I was greeted by the warm hospitality of the locals and the sweet
aroma of freshly caught fish.
\end{spverbatim}

\subsubsection{LLaMA-3-8B Instruct Output}
\begin{spverbatim}
Island Magic: Unveiling the Hidden Gems of Hawaii

As I stepped off the plane in Honolulu, the warm tropical air
enveloped me, and the sweet scent of plumeria flowers filled my
senses. My recent trip to the Hawaiian Islands was a dream come
true, and I'm excited to share with you the incredible experiences
and must-see attractions that left me enchanted.

A Cultural Immersion in Honolulu

My journey began in Honolulu, the capital city of Oahu, where I
immersed myself in the rich cultural heritage of the islands. I
visited the Bishop Museum, the largest museum in Hawaii, which
showcases the history and traditions of the native Hawaiian people.
The museum's collections include an impressive array of artifacts,
including ancient Hawaiian tools, traditional clothing, and even a
replica of a 19th-century Hawaiian village.

Next, I headed to the Honolulu Fish Auction, where I witnessed the
vibrant local fish market in action. The sights, sounds, and smells
of the bustling market were a true sensory experience, and I was
fascinated by the variety of fresh seafood on display.
\end{spverbatim}


\section{Jensen-Shannon Distance Results}
\label{sec:jsd-results}

See Table \ref{tab:jsd_corpora}.

\begin{table}[thbp!]
\centering
\begin{tabular}{lll}
\hline
\textbf{Corpus 1} & \textbf{Corpus 2} & \textbf{JSD} \\
\hline
Nigeria & LLaMA2-base & 0.6913 \\
Nigeria & LLaMA2-chat & 0.6946 \\
US & LLaMA2-base & 0.7062 \\
US & LLaMA2-chat & 0.7128 \\
Nigeria & LLaMA3-base & 0.6915 \\
Nigeria & LLaMA3-chat & 0.7091 \\
US & LLaMA3-base & 0.7064 \\
US & LLaMA3-chat & 0.7224 \\
Nigeria & US & 0.6617 \\
LLaMA2-base & LLaMA2-chat & 0.6486 \\
LLaMA3-base & LLaMA3-chat & 0.6691 \\
LLaMA2-base & LLaMA3-base & 0.6062 \\
LLaMA2-chat & LLaMA3-chat & 0.6011 \\
\hline
\end{tabular}
\caption{Jensen–Shannon distance (JSD) between different corpora. Values rounded to 4 significant figures.}
\label{tab:jsd_corpora}
\end{table}



\section{Downloading English corpora}
\label{sec:corpora}

We downloaded the COCA sample from \url{https://www.corpusdata.org/formats.asp}. We chose the linear text random sample provided (0.1\% of the corpus). We excluded the files on spoken word and TV as their dialogue formatting biases the bigram analysis.

We downloaded ICE Nigeria from \url{https://sourceforge.net/projects/ice-nigeria/}.

\end{document}
