\documentclass[11pt]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
\usepackage{dblfloatfix}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}
\usepackage{float}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{WikiLingDiv: a dataset for quantifying digital linguistic diversity using Wikipedia page views}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{Hannes Essfors \\
  TU Wien Informatics, \\
  Favoritenstraße 9-11,\\
  1040 Vienna, Austria  \\
  \texttt{hannes.essfors@tuwien.ac.at} \\\And
  Andreas Baumann \\
Faculty of Philological and Cultural Studies, \\
University of Vienna, Universitätsring
1,\\
1010 Vienna, Austria\\
  \texttt{andreas.baumann@univie.ac.at} \\}

%\author{
%  \textbf{First Author\textsuperscript{1}},
%  \textbf{Second Author\textsuperscript{1,2}},
%  \textbf{Third T. Author\textsuperscript{1}},
%  \textbf{Fourth Author\textsuperscript{1}},
%\\
%  \textbf{Fifth Author\textsuperscript{1,2}},
%  \textbf{Sixth Author\textsuperscript{1}},
%  \textbf{Seventh Author\textsuperscript{1}},
%  \textbf{Eighth Author \textsuperscript{1,2,3,4}},
%\\
%  \textbf{Ninth Author\textsuperscript{1}},
%  \textbf{Tenth Author\textsuperscript{1}},
%  \textbf{Eleventh E. Author\textsuperscript{1,2,3,4,5}},
%  \textbf{Twelfth Author\textsuperscript{1}},
%\\
%  \textbf{Thirteenth Author\textsuperscript{3}},
%  \textbf{Fourteenth F. Author\textsuperscript{2,4}},
%  \textbf{Fifteenth Author\textsuperscript{1}},
%  \textbf{Sixteenth Author\textsuperscript{1}},
%\\
%  \textbf{Seventeenth S. Author\textsuperscript{4,5}},
%  \textbf{Eighteenth Author\textsuperscript{3,4}},
%  \textbf{Nineteenth N. Author\textsuperscript{2,5}},
%  \textbf{Twentieth Author\textsuperscript{1}}
%\\
%\\
%  \textsuperscript{1}Affiliation 1,
%  \textsuperscript{2}Affiliation 2,
%  \textsuperscript{3}Affiliation 3,
%  \textsuperscript{4}Affiliation 4,
%  \textsuperscript{5}Affiliation 5
%\\
%  \small{
%    \textbf{Correspondence:} \href{mailto:email@domain}{email@domain}
%  }
%}

\begin{document}
\maketitle
\begin{abstract}
With the conflation of digital and non-digital spaces, and NLP technologies being integrated into an increasing number of aspects of daily life, linguistic diversity cannot be fully understood without considering how language is used online. While existing models of linguistic diversity typically have relied on speaker numbers or language production, the dimension of diversity in language consumption remains comparatively understudied. To facilitate such research, we introduce WikiLingDiv, an openly accessible dataset for quantifying linguistic diversity in online knowledge retrieval using Wikipedia page views. Our dataset is based on yearly page views of 340 language editions of Wikipedia, aggregated across 239 countries and territories over 10 years (2015-2024). Using the dataset, we illustrate spatial and temporal patterns of digital linguistic diversity, suggesting that diversity has both increased and decreased across countries and regions, while highlighting country-specific dynamics in language usage. We release the dataset as an openly available and easily integrable data resource for researchers in computational linguistics, digital humanities, and the broader social sciences, enabling further work on linguistic variation, digital inequality, and the interaction between language use and digital technology. 

\end{abstract}

\section{Introduction}

In the postdigital world, the lines between digital and non-digital have become blurred, with communication through digital media and tools moving from representing non-standard use of language to a non-optional part of daily interaction in large parts of the world. As a matter of fact, in December 2025, 73.2\% of the global population were internet users and 68.7\% were social media users \citep{statista2025digitalpopulation}, while during Covid-19, digital working became the de facto standard for much of the knowledge-based part of the global economy \citep{Marsh_Vallejos_Spence_2022}.

This raises the question whether traditional models of linguistic diversity as a function of the relative abundance of L1 speakers \citep{Greenberg_1956, Ginsburgh_Weber_2020, Grin_Fürst_2022, Civico} remain sufficient---notably, the bias towards English in the digital space would suggest a large discrepancy between native language and the actual linguistic landscape \citep{Benson_Zeh_Essfors_Fellner_Neidhardt_Baumann_2025}. Instead, an operationalization of linguistic diversity based on language usage might be considered a more direct model of how language facilitates the tasks of, e.g., retrieving and producing information in the process of between-human interaction. While modelling language usage in the non-digital space intuitively appears difficult, it presents itself as fairly straightforward in the digital space due to the extensive datafication it has been subjected to \citep{Couldry_Meijas_2019, Flyverbom_2019}. Notably, such approaches towards linguistic diversity have been taken by, e.g., \citet{Magdy_Ghanem_Musleh_Mokbel_2014, Alshaabi_Dewhurst_Minot_Arnold_Adams_Danforth_Dodds_2021, Hiippala_Hausmann_Tenkanen_Toivonen_2019, Väisänen_Järv_Toivonen_Hiippala_2022} who assess digital linguistic diversity based on language production on social media, specifically Twitter/X and Instagram. These approaches come with three main caveats: Firstly, social media data is highly sensitive and thus restricted in its usage. For instance, since 2023, the Twitter API is no longer freely available for research \citep{Blakey_2024}. Secondly, using social media data and similar artifacts to model digital linguistic diversity is crucially reliant on high accuracy in the NLP-task of language identification, which is notably challenging for multilingual documents and short texts \citep{Jauhiainen_Lui_Zampieri_Baldwin_Lindén_2019}, both of which are common features of social media posts \citep{Alshaabi_Dewhurst_Minot_Arnold_Adams_Danforth_Dodds_2021}. Thirdly, it has been shown that only a small fraction of tweets are geotagged \citep{pfeffer2023justdaytwittercomplete}, severely limiting the possibility of analyzing linguistic diversity as a spatial phenomenon. 

To address some of these shortcomings and facilitate research on digital linguistic diversity as an approach towards modelling human culture and society, we here introduce \textit{WikiLingDiv}---a dataset for quantifying digital linguistic diversity using Wikipedia page views. Our approach utilizes the fact that Wikipedia is a multilingual, widely used reference work, freely available in most of the world, and as of August 2025, the 7th most visited website globally \citep{statista2025_mostvisited}. Furthermore, the Wikimedia Foundation, which hosts Wikipedia, is committed to open data principles, making records and web activity associated with Wikipedia openly available through the Wikimedia Analytics API. This allows us to collect the number of page views associated with any given language edition of Wikipedia for 239 countries and territories over 10 years, which are made openly accessible through a CC BY-SA 4.0 license. Using this as a basis, we then normalize the page views to yield the proportion of attention given to each language across countries and years, based on which formal measures of diversity are derived.

Explicitly, our approach aims to shed light on digital linguistic diversity as a dynamic phenomenon arising when humans interact with---and through---digital technology. It follows that our work builds on two key assumptions: 1) Wikipedia page views approximate interest in retrieving knowledge in a given language, and 2), Patterns of Wikipedia access are impacted by socioeconomic factors such as internet access and level of education. Therefore, WikiLingDiv does not model digital linguistic diversity as a product of textual output, but rather as a proxy for language consumption. Thus, it complements work focusing on diversity in existing language technology infrastructure, such as the \textit{Digital Language Equality Metric} (DLE) \citep{Gaspari_Gallagher_Rehm_Giagkou_Piperidis_Dunne_Way_2022, Grutzner-Zahn_Rehm_2022} of the European Language Equality (ELE) project, which assesses the availability of technology and support for different languages. Inherently, these two dimensions of digital linguistic diversity are linked: the language used depends on the available technological support for that language, and the available support partially reflects the linguistic demand. To understand the extent of this interaction, data on digital linguistic usage and the relative demand of digital technology is necessary, a gap which WikiLingDiv seeks to help to address.

The paper is structured as follows. First, we describe the process of compiling and structuring the dataset, as well as the measures taken in harmonizing the data to allow for integration with other datasets. Then, we provide a descriptive analysis of the dataset. Finally, we showcase how WikiLingDiv can be utilized in a diversity analysis to address the research question of how digital linguistic diversity---from the perspective of language consumption in online knowledge retrieval---has changed over the last decade.


\section{Creating the dataset}
For the creation of the dataset and all subsequent analysis, the statistical programming language R (4.5.2) \cite{R} was used.\footnote{All code associated with the paper is made available at \url{https://github.com/Eszettfors/WikiLingDiv}. The dataset is published at \url{https://zenodo.org/records/18526766}} As alluded to in the introduction, our empirical strategy utilizes the Wikimedia Analytics REST API to understand how users interact with the language-specific Wikipedia editions.\footnote{For the exact API documentation provided by the Wikimedia Foundation, see \url{https://doc.wikimedia.org/generated-data-platform/aqs/analytics-api/}} More specifically, we use the httr package \cite{httr} to acquire a list of all available Wikimedia projects through the following API-request: \url{https://commons.wikimedia.org/w/api.php?action=\newline sitematrix\&smtype=language\&format=json}. We loop through the list and collect the 2 or 3-letter language codes used by Wikimedia to denote each project, e.g., /en/ for English or /ceb/ for Cebuano, together with its name in Latin script. For each language code, we then query the REST API to acquire the page views per country. The endpoint takes requests according to the following structure: /pageviews/top-by-country/{project}/{access}/{year}/{month}. We specify \textit{project} using the respective language codes; we specify \textit{access} to 'all-access' to capture both desktop and mobile users; and we specify the time frame as each year and month combination beginning with 2015, since the endpoint serves data from this year and onwards. 

From each request to the API endpoint, we receive a value for the number of page views from each country for which at least one pageview exists in a given year and month. However, to protect user privacy, the Wikimedia Foundation does not publish the exact number of page views but instead ceils to the nearest 1000 \cite{wikitechAQSPageviewsPerProject2024}, meaning that the values are estimates. For example, if two language editions receive 1012 and 1756 views, respectively, they are both assigned the value 2000. It naturally follows that the more views an edition has, the smaller the rounding error proportionally is. We then aggregate the values for each year from 2015 to 2024 by summing the page views for each language edition and country across all 12 months. Finally, we restructure the dataset as a time series in long format such that each record consists of a uniquely defined year-country-language triplet specified by the number of page views.

Unfortunately, the page views by country endpoint does not allow specification of the nature of the agent, and accordingly, we cannot purposefully exclude self-identified bots from the statistics. While this naturally introduces some bias to the data, systematic scraping constitutes a fairly stable bias within countries---especially given the low granularity scope the data is aggregated at---meaning that the relative abundances we are seeking to compute is expected to remain robust. Furthermore, we would argue that a substantial fraction of automated information retrieval is indeed driven by human information needs. With this in mind, we suggest the aggregated page views per language and country be interpreted as a proxy for the relative interest in a particular language in that country.

\subsection{Data harmonization}
To harmonize and make the dataset interoperable, we address the language codes used by Wikipedia, which inconsistently employ both three- and two-letter language codes as well as multiple non-standard identifiers \cite{wikimedia-list-of-wikipedias}. There are two major language catalogues that we intend to make the dataset easily mergeable with: Glottolog \cite{glottolog5_2_2025} for linguistic typology and Ethnologue \cite{ethnologue28_2025_online} for speaker numbers. For this purpose we add both ISO 639-3 codes \cite{ISO639_2023} and glottocodes. Since ISO-codes can denote both macrolanguages and individual languages, such as /ara/ being the macrolanguage identifier encompassing all Arabic varieties---multiple of which have their own Wikipedia language edition---we avoid using macrolanguage identifiers, unless they have a corresponding Glottocode. This is, for example, the case with Serbo-Croatian (hbs), which is recognized as an individual language in Glottolog with the respective standardized varieties Bosnian (bos), Serbian (srp) and Croatian (hrv), which are classified as dialects in Glottolog and thus have a corresponding Glottocode. As a result, both Glottocodes and ISO639-3 function as unique language identifiers in the dataset, allowing for easy integration with, e.g., knowledge-bases used in multilingual NLP such as URIEL+ \citep{khan-etal-2025-uriel}.

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{violin_box_ts.png}
    \includegraphics[width=1\linewidth]{Lang_map_pos.png}
    \includegraphics[width=1\linewidth]{dom_lang_map.png}
    \caption{a) Boxplots overlayed by violinplots depicting the distribution of yearly total page views per country from 2015 to 2024, scaled using a base-10 logarithm. Each point corresponds to a country or territory, which is
    coloured according to its geographical continent of belonging. b) Geographical distribution of languages present in the dataset, coloured according to language family. The coordinates of each language were taken from Glottolog. c) World map with each country coloured according to the most visited language edition of Wikipedia between 2015 and 2024. Not all languages are labeled due to overlap. Colour intensity corresponds to the dominance of the language.}
    \label{fig:violin_map}
\end{figure*}


\section{Descriptive analysis}
All in all, the dataset comprises 295,510 year-language-country triplets covering a total of 340 languages and 239 countries/territories across 10 years (2015 - 2024). As seen in Figure \ref{fig:violin_map}a), the distribution of page views across countries remains fairly stable throughout the time period. While a noticeable increase in median and mean page views can be observed from 2015 (mean = $10^{7.38}$; median = $10^{7.37}$ to 2016 (mean = $10^{7.66}$; median = $10^{7.57}$), only small fluctuations are distinguished until 2022, after which it decreases. Furthermore, the geographical spread of the page views is by no means homogeneous: the boxplots are consistently layered according to continents across the time span, with countries in Oceania and Africa generally making up the first and second quartiles; North America the second and third; and Europe, Asia and South America the third and fourth. The number of countries also remains stable around 235, with a sharp decline to only 200 countries in 2024. This can reasonably be assumed to be a result of the Wikimedia Foundation's data publication guidelines, as the missing countries for 2024 align with the list of countries and territories designated as protected due to the higher risk posed to individuals from their Wikipedia activity \cite{wikimedia_country_territory_protection_list_2025}. Because of this, the dataset only contains complete time series for 196 countries and territories, which, however, increases to 232 if the year 2024 is excluded.


\begin{table}[ht]
\centering
\begin{tabular}{lcc}
  \hline
\textbf{Country} & \textbf{Page Views (\%)} & \textbf{Cumulative} \\ 
  \hline
U.S. & 22.50 & 22.50 \\ 
 Japan & 7.06 & 29.56 \\ 
 Germany & 6.35 & 35.91 \\ 
  U.K. & 5.58 & 41.49 \\ 
  India & 4.82 & 46.31 \\ 
  France & 4.03 & 50.34 \\ 
  Italy & 3.62 & 53.95 \\ 
  Russia & 3.44 & 57.39 \\ 
  Canada & 2.76 & 60.16 \\ 
  Spain & 1.99 & 62.15 \\ 
   \hline
\end{tabular}
\caption{Top 10 countries as a source of Wikipedia page views across the years 2015 - 2024.}
\label{tab:cont_pw}
\end{table}

Naturally, Wikipedia is not used to the same extent across countries due to differences in, e.g., population size, digital access, information needs, and government restrictiveness on information access. This is evident from the fact that 22.5\% of total page views across all years stems from the United States, with 62.15 \% of all page views being sourced from 10---predominantly highly developed---countries as seen in Table \ref{tab:cont_pw}. This difference is then partially reflected in which language editions receive attention, with 88.61 \% of attention directed towards the 10 language editions with the most page views, as seen in Table \ref{tab:lang_pw}, largely corresponding to the national language of the countries in Table \ref{tab:cont_pw}. Although the dataset comprises 340 languages in total, almost 50 \% of page views are concentrated into a single language, English, and looking at a family level, of the 32 language families present, 87.74 \% of attention is directed towards the 137 Indo-European languages in the dataset, of which English is one. Indeed, as can be seen in Figure \ref{fig:violin_map}b), most languages are found across Eurasia, with a particularly high concentration of languages in Europe. However, quite a few languages from Central Asia, especially the Caucasus, are present. Also in Africa, notably the Atlantic-Congo (42 languages) and Afro-Asiatic (16 languages) language families are present. Languages from the Americas and Oceania, however, have a noticeably low presence in the data.

\begin{table}[ht]
\centering
\begin{tabular}{lcc}
  \hline
  \textbf{Language} &  \textbf{Page Views (\%)} & \textbf{Cumulative} \\ 
  \hline
English & 49.77 & 49.77 \\ 
 Japanese & 6.92 & 56.69 \\ 
 Spanish & 6.55 & 63.24 \\ 
 German & 5.96 & 69.21 \\ 
 Russian & 5.07 & 74.28 \\ 
 French & 4.66 & 78.94 \\ 
 Italian & 3.38 & 82.32 \\ 
 Chinese & 2.67 & 84.99 \\ 
  Portuguese & 2.02 & 87.00 \\ 
 Polish & 1.60 & 88.61 \\  
   \hline
\end{tabular}
\caption{Top 10 visited language editions of Wikipedia over 2015 and 2024.}
\label{tab:lang_pw}
\end{table}

On the other hand, looking at the country level and the dominant language of each country as seen in Figure \ref{fig:violin_map}c), only 24 languages dominate the 239 countries and territories in the data. While many instances of the national language being the most dominant language are found in Europe, such as for Bulgaria, Sweden and Finland, a pattern of imperialistic legacy can be observed. For example, West Africa is dominated by French, while East Africa is dominated by English. Portuguese is dominant in the former colonies of Angola, Mozambique and Brazil, while the Russian Wikipedia is most popular in former Soviet states such as Ukraine and Kazakhstan. However, Russian is clearly more dominant in Russia (88\% of page views) compared to Ukraine (57\%) and Kazakhstan (66\%), signaling that there is competition between Russian on the one hand, and the local national languages of Ukrainian (30\%) and Kazakh (25\%) and to a certain extent English (9.8\% \& 6.5\%) on the other hand. At the same time, in other former Soviet states, such as Armenia (40\% Armenian, 36 \% Russian, 20\% English) and Azerbaijan (42\% Azerbaijani, 25\% Russian, and 18\% English), the local national languages are dominant, with the Russian Wikipedia being the second most used language. In the Baltic states, English dominates instead, followed by the local national language and Russian (Estonia: English 43\%, Estonian 35\%, Russian 16\%; Latvia: English 37\%, Russian 31\%, Latvian 24\%; Lithuania: English 43\%, Lithuanian 41\%, Russian 11\%). Reasonably, these patterns occur as a result of different pressures acting on internet users, such as knowledge and familiarity of the languages in the population, the extent and usefulness of the Wikipedias \cite{kornai2013digital}, and the political status and association with the languages, to name a few; all factors that suggest themselves to be accounted for in follow-up research. We are thus convinced that our dataset has great potential as a resource to model language competition and further our understanding of how technology interacts with---and shapes---both human society and cultural heritage. 

\section{Analyzing change in linguistic diversity}
In this final part of the paper, we showcase how our dataset can be employed to answer research questions pertaining to linguistic diversity. Specifically, we aim to understand if digital retrieval of information has become more or less linguistically diverse over the last decade. Since the country coverage is substantially lower for 2024, we restrict the analysis to the years from 2015 to 2023, giving a time series of nine consecutive years. To increase the reliability of our estimates, we filter away countries that in any given year had fewer than 1,000,000 page views in total, which we consider a reasonable threshold. We then subset to countries and territories present across all nine years, yielding a complete time series with 199 countries. As a strategy to protect against sudden spikes in language richness due to scraping activities, we fixate the languages in each country to those present in 2015. Therefore, any changes in diversity will reflect a redistribution of attention among already established languages, rather than the introduction of new languages in the respective countries. While this strategy runs the risk of missing changes related to large-scale migration that explicitly took place after 2015, the actual impact on the measured diversity is likely limited within the relatively short timespan under study.

For each year, we calculate a relative abundance vector for each country by dividing the number of page views of each language by the total number of page views in that country. As such, we end up with $232 \times 9$ vectors, $p = [p_1, p_2, ..., p_n]$ where $p_n$ is the proportion of page view visits to the n:th Wikipedia, in a given country and year, such that

\begin{equation}
    \sum_{i=1}^Rp_{i} = 1,
\end{equation}
\noindent
where $R$ is the length of the relative abundance vector.

We then define linguistic diversity according to the Leinster Cobbold Framework \cite{LeinsterCobbold2012}---suggested by \citet{Essfors2025GlobalLinguisticDiversity}---as the effective number of completely dissimilar and equally abundant languages. While we acknowledge that diversity is multivariate in its nature, we only employ one measure of diversity to maintain the example scope of the analysis. Specifically, we use a naive model of diversity, treating all languages as completely dissimilar and distinct from one another. To reduce the influence of rare languages that might have been introduced through automatic scraping and not necessarily purposeful information retrieval, we specify the sensitivity parameter $q$ to 2, such that rare languages are given a lower weighting. In this set-up, our working definition of diversity $D_{tc}$ in the country $c$ at time $t$ formally becomes 
\begin{equation}
    D_{tc} = \frac{1}{\sum_{i=1}^Rp_i^{q}}, q = 2 .
\end{equation}
\noindent
First, we consider the absolute change in diversity from 2015 to 2023 by calculating the log diversity ratio between the years 2015 and 2023 as an effect size. Formally, we define the log diversity ratio $ldr$ for a given country $c$ as
\begin{equation}
    ldr(c) = log_2(\frac{D_{2023c}}{D_{2015c}})
\end{equation}
\noindent
where $D_{2023c}$ and $D_{2015c}$ are the measured diversity of $c$ in 2023 and 2015 respectively. It follows that a positive log-ratio indicates an increase in diversity, and a negative log-ratio a decrease. Furthermore, the measure is symmetric around $0$ (i.e., no change), such that a doubling of diversity = $1$, while a halving of diversity = $-1$.

\begin{figure}
    \centering
    \includegraphics[width=1\linewidth]{plots/log_div_ratio.png}
    \caption{A barchart displaying the change in diversity across countries between 2015 and 2023. Each bar represents a country, with the length of the bar being proportional to the change in diversity. Bars directed to the left indicate a decrease in diversity, while bars directed to the right indicate an increase.}
    \label{fig:log_ratio_bar}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=1\linewidth]{steamgraph.png}
    \caption{A streamgraph showing the proportional trajectories of dominant languages in Brazil, Iraq, Tanzania und Uzbekistan from 2015 to 2023.}
    \label{fig:streamgraph}
\end{figure}

By plotting the $ldr$ of each country as seen in Figure \ref{fig:log_ratio_bar}, it is evident that more countries have seen an increase in diversity than a decrease. As a matter of fact, we note an increase in 137 countries, more than twice the number of countries observing a decrease in diversity (62), yielding a mean $ldr$ of 0.095, 95\% CI = [0.063, 0.13], suggesting that on average, linguistic diversity has increased by approximately 4.5\%--9.4\% from 2015 to 2023 due to a redistribution of attention away from the most dominant languages within each country. One of the most important insights from this result is that while indeed diversity is suggested to generally be increasing, the fact that the change is heterogeneous---with diversity decreasing for a substantial portion of countries---indicates that the underlying mechanisms are country-specific, and potentially culturally dependent. Considering the spatial distribution of $ldr$ as seen in Figure \ref{fig:log_ratio_map}, there is clearly some spatial dependence, with a substantial increase in diversity across most of Latin America, Central and South Asia, while the countries of the Middle East have seen a reduction. By investigating the relative abundance trajectories of the dominant languages within the respective countries, we can discern different patterns of interplay between the competing languages.

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{plots/log_ratio_map.png}
    \caption{A coropleth map visualizing country-level change in the log diversity ratio (ldr) from 2015 to 2023. A blue color indicates an increase in diversity while a red color indicates a decrease, with more intense colors corresponding to a larger magnitude of change. An ldr of 1 is equivalent to a doubling of diversity. Grey countries indicate missing values.}
    \label{fig:log_ratio_map}
\end{figure*}

For example, across Latin America, exemplified by Brazil in Figure \ref{fig:streamgraph}, we see a clear dominance of the national language---in the Brazil case, Portuguese---which has been declining in recent years, with increasing attention given to English, resulting in a diversification. This highlights an important aspect of diversity: what is perceived as diversification on the country level can be the result of a process yielding a concentration of languages on a global level. English is by far the most widely used language digitally, and as such an increase in its usage represents a decline in global diversity, while on a local level---as in Brazil---it is perceived as a diversification.

In Iraq and across the Middle East, on the other hand, the reverse pattern is observed: the English and standard Arabic wikis were on equal footing in 2015, after which increasing attention has been given to the Arabic one, making it dominant, thus resulting in a measured decline in diversity locally. This would, however, contribute to a global increase in diversity. In Tanzania, a similar pattern is observed, with English in steady decline and attention being directed towards the Wikipedia in the local language of Swahili. And since English was heavily dominating in 2015, this resulted in an increase in measured diversity. In Uzbekistan, a third pattern is observed, with English remaining fairly constant across time, while the dominant Russian language loses ground to the national language of Uzbek, which only constituted 5.1\% of page views in 2015, gradually increasing its share to 40.0\% in 2023, just barely behind Russian with 40.1\%.

While we can use WikiLingDiv to describe these patterns in language attention, the data in itself cannot explain why they occur. For this reason, other dimensions of digital linguistic diversity are equally important to consider, e.g., how the digital support for Arabic, Swahili and Uzbek has changed relative to English and Russian. The addition of such data could yield insights into how extending the technological resources of a language impacts its usage. For this purpose, intricate measures of language support, such as that of the DLE \citep{Gaspari_Gallagher_Rehm_Giagkou_Piperidis_Dunne_Way_2022, Grutzner-Zahn_Rehm_2022} for European languages, or Digital Language Support (DLS) of \cite{simons2022assessingdigitallanguagesupport}, building on \cite{kornai2013digital}, for global estimates. To extend the analysis presented here, the most straightforward way would be to construct a metric quantifying the extent of the respective Wikipedia language edition, using, e.g., the number of Wikipedia articles and their sizes. However, equally important would be to also consider socioeconomic factors such as education level and second language acquisition, e.g., the increased use of the English Wikipedia in Latin America could very well be reflective of an increase in English proficiency. Further analyses could for example include the EF English Proficiency Index as a covariate to account for such heterogenity. 

\section{Discussion and Conclusion}
In this paper, we have introduced WikiLingDiv, a dataset for quantifying linguistic diversity in the digital space using Wikipedia page views as a proxy for interest in retrieving information in different languages. We have showcased its extensiveness with respect to spatial and temporal coverage, as well as linguistic breadth. In doing so, we illustrated how previously observed linguistic biases in available digital content---to a degree---manifest as patterns of digital knowledge retrieval. However, information retrieval and digital language consumption is only one dimension of digital linguistic diversity, and further research should aim at combining the dataset with other operationalizations, such as digital language support metrics.

While our approach is somewhat limited by the inability to differentiate between human and bot-generated page views, we have exemplified strategies to mitigate the impact of automatic scraping while still capturing signals of human linguistic preference. Furthermore, the dataset is potentially limited by the use of translation tools; it could very well be that traffic is directed towards the extensive English wikipedia, but then machine translated into a different language. This potentially yields a bias where the dominance of English is overestimated and the attention to languages with machine translation support is underestimated. In addition, while Wikipedia is one of the most widely used websites globally, its usage can be expected to vary within and across populations; e.g., men use Wikipedia to a larger extent than women \citep{Johnson_Lemmerich_Saez-Trumper_West_Strohmaier_Zia_2021}, there is a notorious rural-urban divide in internet access \citep{ITU2024UrbanRuralInternet}, and in some countries, access to Wikipedia has purposefully been restricted by the government \cite{Zhang_Livneh_Budak_Robert_Romero_2017, Pan_Roberts_2020, Yang_Roberts_2021}. Still, Wikipedia is arguably the closest approximation to a global and universilly used digital encyclopedia there is.

Through our example analysis, we have demonstrated how attention towards different language editions of Wikipedia varies over time and across countries, resulting in cases of measurable increase as well as decrease in linguistic diversity. Most strikingly, however, we observe signs of an overall tendency towards \textit{increasing} linguistic diversity, as per our operationalization, which to a certain extent challenges the widely-spread notion that linguistic diversity is steadily declining \citep{harmon2010indexhdl, Bromham_2023}. However, since the increase is an average across countries, the observed diversification could result from English increasing its relative usage compared to already dominating language such as Spanish and Portuguese. Importantly, the longitudinal and spatial resolution of the data enables the identification of regional patterns of attention shifts from dominating global languages, such as English, to local languages---an interaction that ultimately plays a central role in shaping digital linguistic diversity. Explaining why these patterns emerge requires extending the analysis to inferential models, accounting for, e.g., the digital support of the languages, extent of internet access and digitalization across countries,  and the linguistic diversity in the non-digital linguistic landscape, which could yield valuable insights into how diversity in digital language usage is shaped by the linguistic diversity of technology.

Ultimately, understanding the interaction between language, technology, and culture is crucial as societies become increasingly digital, and NLP tools are further integrated into everyday life. While this endeavor is inherently complex, WikiLingDiv is a reusable and easy-to-integrate dataset that can support further research in areas of study where NLP, digital humanities, and linguistic diversity meet.

\section*{Acknowledgments}

This research was funded by WWTF (grant number ICT23-012).

\bibliography{custom}

\end{document}
