%
% WAT-2024
% An Empirical Study of Multilingual Vocabulary for Neural Translation Models
%
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
%\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

%\documentclass[11pt]{article}
\documentclass[11pt,dvipdfmx]{article}

% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
%\usepackage[review]{acl}
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}

%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

%
% My settings
%
%\usepackage[dvips]{graphicx}
\graphicspath{{./fig/}}
\usepackage{multirow}
\usepackage{amssymb}
\usepackage{booktabs,array}
%\usepackage{jumoline}

%%% fractions
\setcounter{topnumber}{3}
\setcounter{dbltopnumber}{3}
\renewcommand{\topfraction}{0.9}
\renewcommand{\dbltopfraction}{0.9}
\renewcommand{\floatpagefraction}{0.9}
\renewcommand{\dblfloatpagefraction}{0.9}
\renewcommand{\textfraction}{0.2}

%
% Title and authors
%
\title{An Empirical Study of Multilingual Vocabulary \\
for Neural Machine Translation Models}
\author{Kenji Imamura \and Masao Utiyama \\
  National Institute of Information and Communications Technology, \\
  Seika-cho, Kyoto, 619-0289, Japan \\
  \texttt{\{kenji.imamura, mutiyama\}@nict.go.jp}
}

%%%%%%%%%%%%%%%%
\begin{document}
\maketitle

%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
In this paper, we discuss multilingual vocabulary for neural machine
translation models.  Multilingual vocabularies should generate highly
accurate machine translations regardless of the languages, and have
preferences so that tokenized strings contain rare out-of-vocabulary
(OOV) tokens and token sequences are short.  In this paper, we discuss
the characteristics of various multilingual vocabularies via
tokenization and translation experiments.  We also present our
recommended vocabulary and tokenizer.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\label{sec:introduction}
%%%%%%%%%%%%%%%%%%%%%

In recent tasks that use neural models, including neural machine
translation, we usually fine-tune pretrained models (e.g.,
\citet{devlin-etal-2019-bert,liu-etal-2020-multilingual-denoising}).
When a pretrained model is fine-tuned, the training corpora are
different from those used for pretraining, in which the vocabulary
must be different.  However, pretrained models determine their
vocabulary in advance, and it is difficult to change the vocabulary
during fine-tuning.  Therefore, it is important to discuss the first
vocabulary.%
\footnote{It is possible to only add words in the vocabulary 
\cite{Tang:mbart50-2020,imamura2022extendingsubwordingmodelmultilingual}.}

On the other hand, it becomes common to process multiple languages in
machine translation and large language models (LLMs) because neural
models can be packed multiple languages into a model (e.g.,
\citet{johnson-etal-2017-googles}).
%
In this paper, we discuss vocabularies appropriate for multilingual
neural models.  The target task is machine translation that uses
encoder-decoder models.  Our aim is to decide the vocabulary that is
suitable for our multilingual translation models.

Figure \ref{fig:vocabulary-modules} illustrates the typical structure
of an encoder-decoder model
\cite{DBLP:journals/corr/VaswaniSPUJGKP17}.  In this structure, there
are five modules related to vocabulary: 1) source tokenizer, 2) target
tokenizer, 3) encoder embeddings, 4) decoder embeddings, and 5) output
projector.
%
The tokenizers tokenize a string into tokens, which consist of
(sub-)words in the vocabulary of each tokenizer, except for
out-of-vocabulary (OOV) strings.  Neural models convert them into
dense representations by looking up the tokens in the word embedding
tables.  Thus, the vocabularies in the tokenizers and neural model
(the embedding tables and output projector) are essentially identical.
%
It is possible to use different vocabularies between the encoder and
decoder.  However, shared vocabulary is generally used in multilingual
models because both input and output strings are multilingual (e.g.,
\citet{liu-etal-2020-multilingual-denoising,https://doi.org/10.48550/arxiv.2010.11125}).
In this paper, we assume that the vocabularies of the above five
modules are identical, unless otherwise specified.

%%%%%%%%%%%%%%
\begin{figure}
\begin{center}
\includegraphics[clip,width=\linewidth]{fig-encoder-decoder.eps}
\end{center}
\caption{Vocabulary-related modules in an encoder-decoder model.}
\label{fig:vocabulary-modules}
\end{figure}
%%%%%%%%%%%%%%

We suppose that the preferences or requirements of the multilingual
vocabulary for neural models are as follows.

\begin{enumerate}
\item
High accuracy is preferred in target tasks.  Because we use the machine
translation task in this paper, high translation quality is preferred.
\label{enum:trans-quality}

\item
Token sequences, into which arbitrary strings are tokenized using the
vocabulary, do not contain OOV tokens.  This is a high preference
because the OOV tokens certainly reduce the accuracy of tasks
\cite{sennrich-etal-2016-neural}.
\label{enum:no-unk}

\item
Token sequences are short (i.e., the numbers of tokens are small)
because, generally, the shorter the input, the better the output
\cite{arivazhagan2019massivelymultilingualneuralmachine}.
\label{enum:num-of-tokens}

\item
Small models (i.e., the number of model parameters is small) are
better for computation during training and inference.  The number of
parameters in the word embedding tables increases in proportion to the
vocabulary size and accounts for a large portion in neural models.
Therefore, a small vocabulary size is better from the viewpoint of the
number of model parameters.  However, it results in longer token
sequences, and a trade-off emerges between it and a preference for
No. \ref{enum:num-of-tokens}.  We determine the balance of the two
preferences using translation quality.

\item
Regardless of the languages, strings with the same meaning are
tokenized into similar numbers of tokens.  We presume that this
preference reduces complexity during translation.

\item
The token sequences can be read by humans.  Although this preference
does not affect translation quality, high readability is better for
debugging by humans.
\label{enum:visiblity}
\end{enumerate}

In this paper, we discuss the vocabularies that satisfy the above
preferences for multilingual models, which manage a mixture of various
script types.
%
Note that we consider No. \ref{enum:trans-quality} to be the most
important preference, the second preference is No. \ref{enum:no-unk},
and the remaining preferences are optional.

The remainder of this paper is organized as follows: In Section
\ref{sec:related-work}, we explain related work, which includes
studies of multilingual models.  Next, we discuss preferred vocabulary
via tokenization and translation experiments in Sections
\ref{sec:tokenization-exp} and \ref{sec:translation-exp},
respectively.  In Section \ref{sec:vocab-studies}, we compare our
experimental results with findings of conventional vocabulary studies,
and we conclude the paper in Section \ref{sec:conclusions}.

%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}
\label{sec:related-work}
%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%
\subsection{Multilingual Models}

%%%%%%%%%%%%%%
\begin{table*}
\begin{center}
{\small
\begin{tabular}{lllrrc}
\toprule
Type 
& Model
& Tokenizer
& \#Langs.
& Vocab. size
& Byte fallback \\
\midrule

Encoder only
& mBERT & WordPiece & 104 & 120K & \\
& & \cite{WordPiece2012} & & & \\

& XLM-R$\dag$ & SentencePiece/Unigram & 100 & 250K & \\
& & \cite{kudo-richardson-2018-sentencepiece} & & & \\
\cmidrule{1-6}

Encoder-decoder
& mBART$\dag$ & SentencePiece/Unigram & 100 & 250K & \\
& M2M-100 & SentencePiece/BPE & 100 & 128K & \\
& NLLB-200 & SentencePiece/BPE & 200 & 256K & \\
& mT5 & SentencePiece/Unigram & 101 & 250K & $\checkmark$ \\
\cmidrule{1-6}

Decoder only
& GPT2 & Byte-level BPE & 1 & 50K & $\checkmark$ \\
& & \cite{radford2019language} & & & \\
& LlaMa2 & SentencePiece/BPE & 1+$\ddag$ & 32K & $\checkmark$ \\
\bottomrule
\end{tabular}
}
\caption{Tokenizer and vocabulary of major multilingual models.
  $\dag$XLM-R and mBART use the same tokenizer with the same
  vocabulary.  $\ddag$ 90\% of the training corpus of LlaMa2 is in
  English, and the rest is multilingual.}
\label{fig:existing-tokenizers}
\end{center}
\end{table*}
%%%%%%%%%%%%%%

Table \ref{fig:existing-tokenizers} shows the list of major
multilingual (partially monolingual) models and their
vocabularies/tokenizers.

Multilingual BERT (mBERT) \cite{devlin-etal-2019-bert} and
XLM-RoBERTa (XLM-R) \cite{conneau-etal-2020-unsupervised} are
categorized as multilingual encoder models.  These encoder models are
applied to various natural language understanding tasks.

For encoder-decoder models, which are used for machine translation,
multilingual BART (mBART)
\cite{liu-etal-2020-multilingual-denoising,Tang:mbart50-2020},
M2M-100 \cite{https://doi.org/10.48550/arxiv.2010.11125}, NLLB-200
\cite{nllbteam2022language}, and mT5 \cite{xue-etal-2021-mt5} are
categorized as the multilingual models.  Note that mBART and XLM-R use the
same tokenization model.

Recent LLMs are resultantly multilingual, even though they learn using
English Web text, because they contain other languages.  Their
vocabulary sizes are rather small: the size of GPT2
\cite{radford2019language} is 50K and that of LlaMa2
\cite{touvron2023llama} is 32K.

Many multilingual models use SentencePiece
\cite{kudo-richardson-2018-sentencepiece} as their tokenizers.  In
this paper, we use SentencePiece for our experiments.
%
Note that byte pair encoding (BPE) \cite{sennrich-etal-2016-neural}
and unigram models \cite{kudo-2018-subword} are known as major subword
encoding methods.  We use the unigram models in this paper.

%%%%%%%%%%%%%%%%%%%%%
\subsection{Byte-level BPE / Byte Fallback}

%%%%%%%%%%%%%%
\begin{table}
\begin{center}
{\small
\begin{tabular}{lp{43mm}}
\toprule
Method
& Example \\
\midrule

Source
& \verb*|群衆が集結しました。|
\\

1) Character 
& \verb*|群 <UNK> が 集 結 しました 。|
\\

2) Byte fallback
& \verb*|群 <0xE8> <0xA1> <0x86> が 集 結 しました 。|
\\
\bottomrule
\end{tabular}
}
\end{center}
\caption{Example of byte fallback. Japanese character `衆' is
  fallbacked if it is not contained in the vocabulary.}
\label{tbl:byte-fallback}
\end{table}
%%%%%%%%%%%%%%

If an input string contains OOV characters, there are two behaviors
of tokenizers (Table \ref{tbl:byte-fallback}).

\begin{enumerate}
\item[1)]
The tokenizer decomposes the OOV parts into characters.  In this case, the
word embeddings become unknown (indicated by \texttt{<UNK>}).

\item[2)]
The tokenizer decomposes the OOV parts into byte sequences
\cite{radford2019language}.  This method is called byte-level BPE in
the byte-pair encoding and byte fallback in SentencePiece.  They
assume that input strings are encoded in UTF-8.
%
If the vocabulary of the neural models includes all bytes (256 bytes),
no OOV tokens occur.  However, readability decreases because humans
cannot understand the string.
%
Additionally, the decoder may generate invalid byte sequences that are
not decoded into UTF-8 if byte fallback is applied to the decoder.
The detokenizer must address this problem.
\end{enumerate}

We also confirm the effects of byte fallback.

%%%%%%%%%%%%%%%%%%%%%
\subsection{Flores+ Dataset}

The Flores+ dataset \cite{nllbteam2022language,flores101}%
\footnote{\url{https://github.com/openlanguagedata/flores}}
is an evaluation dataset that covers 200 languages.
%
It was created by translating sentences that were sampled
from articles in English Wikinews, Wikijounior, and Wikivoyage into
other languages.  Therefore, the sentences are parallel among
languages other than English.
%
A total of 997 and 1,012 sentences are published as the development
(dev) and development-test (devtest) sets, respectively.%
\footnote{The test set is not published.}

The dataset contains the language and its script type in the
filenames.  We use the categories (language names and script types) of
Flores+ in this paper.

%%%%%%%%%%%%%%%%%%%%%
\section{Tokenization Experiments}
\label{sec:tokenization-exp}
%%%%%%%%%%%%%%%%%%%%%

In this section, we evaluate tokenization using various
vocabularies/tokenizers.  We evaluate translation in Section
\ref{sec:translation-exp}.

%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental Settings}

\paragraph{Target Languages}

We selected 98 languages (26 script types) from the Flores+ dataset
for which there were more than 100K lines in the CC-100 corpus (a set
of monolingual corpora)
\cite{conneau-etal-2020-unsupervised,wenzek-etal-2020-ccnet}.

Considering the script types of Flores+, 55 out of 98 languages use a
Latin script, such as English, and 20 languages use scripts unique to
each language, such as Greek, (simplified and traditional) Chinese,
Japanese, and Thai.
%
The list of languages and script types is shown in Table
\ref{tbl:language-list} in Appendix \ref{sec:language-list}.

\paragraph{Tokenizer/vocabulary}

We evaluated M2M-100, XLM-R/mBART, NLLB-200, mT5, and LlaMa2 for
existing models.  For our original models, we evaluated unigram models
of SentencePiece learned under various conditions.

\paragraph{Training Corpus for SentencePiece}

We randomly selected the training sets for each language from the
CC-100 corpus.%
\footnote{The largest set in CC-100 is 1.8 billion lines of English
  and the smallest set is 120 thousand lines of Lingala.}
%
We selected 20 million lines in total.  The mean number of lines was
approximately 200 thousand per language, but we controlled the sampling
size using a temperature coefficient, as we describe later.

\paragraph{Other Settings for SentencePiece}

We used 0.9995 for character coverage,
and the number of seed pieces was 100 times the vocabulary size.

\paragraph{Evaluation}

We evaluated the tokenization results of the 98 devtest sets in
Flores+ using the following metrics.

\begin{itemize}\itemsep=0mm\parskip=0mm
\item average number of tokens and variance
  (standard deviation) for all languages.
\item total number of OOV tokens.
\item number of fallbacked bytes when we applied byte fallback.
\end{itemize}

\noindent
We preferred a small number of tokens (i.e., short token sequences) and
a small number of OOV tokens.  The low variance of the number of tokens
indicated that sentences with the same meaning were tokenized in close
number of tokens, regardless of the languages.

\paragraph{Comparison Methods}

We compared tokenizers/vocabularies under various conditions as follows.

\begin{itemize}

\item \textbf{Vocabulary Size:} \\
We compared the vocabulary sizes 250K and 64K (or 100K).  The
vocabulary size affects the length of token sequences and neural model
size.

\item \textbf{Byte Fallback:} \\
We compared cases with and without byte fallback.
This condition influences the number of OOV tokens.

\item \textbf{Additional Characters:} \\
We added approximately 52K characters, which are U+0000 to U+D7FF in
the basic multilingual plane of Unicode and have character names in
the Python unicodedata module.  Adding characters to the vocabulary
enables us to control OOV tokens using an alternative to byte
fallback.

Note that we can also control OOV tokens by changing the character
coverage setting during SentencePiece training.  In this study, we
used the additional character method to control OOV tokens.

\item \textbf{Language Balance:} \\
How to determine the sampling size of the training corpus for each
language.  We evaluated the following two methods, one is based on
language distribution in the corpus, and another is based on the
script types. The methods changed the importance of low-resource languages and
languages that use the unique scripts.  Both methods control the
corpus size using the inverse temperature coefficient $1/\tau$
(temperature sampling)
\cite{lample:xlm,arivazhagan2019massivelymultilingualneuralmachine}.

\begin{enumerate}

\item[a)]
This case follows the distribution of the CC-100 corpus (hereafter,
`Corpus').  This means that the size of high-resource languages
becomes large.  The training corpus size $s_{l}$ of language $l$ is
determined by the following equation.
%
\begin{equation}
s_{l} \propto \left( \frac{c_{l}}{\sum_i^L c_i} \right)^{1/\tau},
\end{equation}
%
where $c_{l}$ denotes the number of CC-100 lines of language $l$,
and $L$ denotes the number of languages ($=98$).

\item[b)]
This case uses the script types (hereafter, `Script').  The training
size of each language is uniform for a script type.  Smoothing is
based on the number of languages in a script type.  The size of the
languages that use unique scripts becomes large and that of the
languages using the Latin script becomes small even though we apply
temperature sampling.
%
\begin{equation}
s_{l} \propto \frac{(1/ns_{l})^{1/\tau}}{\sum_i^L(1/ns_i)^{1/\tau}},
\end{equation}
%
where $ns_{l}$ denotes the number of languages in the script type to
which language $l$ belongs (e.g., 55 languages belong to the Latin
script type, and one language belongs to the Japanese script type).

\end{enumerate}
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%
\subsection{Result 1: Language Balance}
\label{sec:result1}

Before the comparison experiments, we determined the optimal inverse
temperature coefficient $1/\tau$ by changing the training corpus size
for SentencePiece.  We evaluated the 250K vocabulary with byte
fallback without additional characters.

%%%%%%%%%%%%%%
\begin{figure}
\begin{center}
\includegraphics[clip,width=\linewidth]{graph-csiz-250k.eps}
a) Corpus: Using the distribution of CC-100.
\vspace*{4ex}

\includegraphics[clip,width=\linewidth]{graph-byte-250k.eps}
b) Script: Using the script types.
\end{center}
\caption{Number of tokens of Flores+ according to the inverse
  temperature coefficient $1/\tau$.}
\label{fig:lang-balance}
\end{figure}
%%%%%%%%%%%%%%

Figure \ref{fig:lang-balance} shows the change of the number of tokens
in the Flores+ devtest set when we changed the inverse temperature
coefficient from 0.0 to 1.0.  It includes the average of all
languages, the average of the languages that use the unique script (20
languages; represented as `Single' languages), and the average of the
languages of the Latin script (55 languages; `Latin' languages).

a) When we used the Corpus method, the number of tokens and
the difference between the Single and Latin languages became the
smallest when $1/\tau = 0.0$.

b) When we used the Script method,
the number of tokens in the Latin languages increased as $1/\tau$
increased.  Conversely, that of the Single languages decreased as
$1/\tau$ increased and they were balanced when $1/\tau = 0.2$.

These results show that it was effective to balance languages by
changing the training corpus size of each language using the inverse
temperature coefficient.  In subsequent experiments, we used the
optimal inverse temperature coefficient that balanced all languages,
that is, the standard deviation of the number of tokens became the
smallest.

%%%%%%%%%%%%%%%%%%%%%
\subsection{Result 2: Tokenization}

%%%%%%%%%%%%%%
\begin{table*}
\begin{center}
{\small
\begin{tabular}{l|lcccccrr}
\toprule
\multicolumn{2}{c}{Tokenizer/}
& Vocab.
& Byte
& Additional
& Lang.
& Avg. \#tokens
& 
& \# Fallbacked \\

\multicolumn{2}{c}{vocabulary}
& size
& fallback
& characters
& balance
& (std. dev.)
& \#OOV
& bytes \\
\midrule

\multicolumn{2}{l}{Baselines} & \\
\cmidrule{2-9}

& M2M-100 & 128K & &
& Corpus
& 42,196 (8,542)
& 38,942
& N/A \\

& mBART/XLM-R & 250K & &
& Corpus
& 37,632 (6,246)
& 30
& N/A \\

& NLLB-200 & 256K & &
& Corpus
& 37,579 (4,900)
& 16,739 
& N/A \\

& mT5 & 250K & $\surd$ &
& Corpus
& 45,365 (9,979)
& 0
& 81 \\

& LlaMa2 & 32K & $\surd$ &
& --\footnotemark
& 96,836 (75,630)
& 0
& 2,989,581 \\
\midrule

\multicolumn{2}{l}{SentencePiece/Unigram} & \\
\cmidrule{2-9}

& 250K\_C+B & 250K & $\surd$ & 0
& Corpus
& 35,562 (3,510)
& 0
& 11,601 \\

& 250K\_S & 250K & & 0
& Script
& 35,900 (3,422)
& 1,873
& N/A \\

& 250K\_S+B & 250K & $\surd$ & 0
& Script
& 35,948 (3,367)
& 0
& 5,848 \\

& 250K\_S+B+C52K & 250K & $\surd$ & 52K
& Script
& 37,095 (3,602)
& 0 & 48 \\

& 64K\_S+B & 64K & $\surd$ & 0
& Script
& 45,504 (4,294)
& 0
& 4,745 \\

& 100K\_S+B+C52K & 100K & $\surd$ & 52K
& Script
& 47,410 (4,676)
& 0 & 48 \\
\bottomrule
\end{tabular}
}
\caption{Tokenization results.  The tokenizer/vocabulary names of
  SentencePiece are combinations of the vocabulary size, language
  balance (`C' and `S' represent `Corpus' and `Script,' respectively),
  byte fallback (`B'), and additional characters (C52K).  }
\label{tbl:tokenization-results}
\end{center}
\end{table*}
%%%%%%%%%%%%%%

Table \ref{tbl:tokenization-results} shows the tokenization results for
the Flores+ devtest set using various tokenizers and vocabularies.
%
`Avg. \#tokens' is the average number of tokens in all languages, and
its standard deviation indicates the variance among languages.  If the
standard deviation is small, differences among languages must also be
small.  `\#OOV' indicates the total number of OOV tokens, and
`\#Fallbacked bytes' is the total number of fallbacked bytes in all
languages.

First, we confirmed the tokenization results of the baselines.  The
mBART/XLM-R and NLLB-200 tokenizers generated the least number of
tokens, and mBART/XLM-R generated the least OOV tokens of the two
tokenizers, even though it does not use byte fallback.  From the
viewpoint of OOV tokens, mT5, which uses byte fallback, was the best;
however, the number of tokens was more than that of mBART/XLM-R.  We
consider that mBART/XLM-R was the most suitable tokenizer/vocabulary
for our preferences (c.f., Section \ref{sec:introduction}).

Next, we compared our SentencePiece unigram models, referring to the
preferences.  We confirm the translation quality in the next section.

First, the number of OOV tokens became zero using byte fallback.

The average number of tokens was most affected by the vocabulary size.
The tokenizers with the 250K vocabulary became a similar number of
tokens regardless of the other conditions.
%
Although not shown in the table, the vocabulary size also affected the
number of model parameters.  When we used a Transformer big model
\cite{DBLP:journals/corr/VaswaniSPUJGKP17}, the number of model
parameters was approximately 430 million for the 250K vocabulary and
240 million for the 64K vocabulary.  The vocabulary size is a
trade-off between the number of tokens and the number of parameters,
and we determined the optimal size using translation quality.

The standard deviation of the number of tokens indicates the variance
of languages.  However, it was less affected by the language balance
and byte fallback because all deviations of the 250K tokenizers were
less than 3,700.  It was most influenced by the size of the training
corpus, as shown in Section \ref{sec:result1}.

Finally, focusing on the number of fallbacked bytes, the number
decreased when there were additional characters.  For example, 5,848
bytes in \texttt{250K\_S+B} decreased to 48 bytes in
\texttt{250K\_S+B+C52K}.  Adding characters is a solution to improve
readability if translation quality is the same.

Tokenization examples of several languages are shown in Tables
\ref{tbl:tokenization-example1} to \ref{tbl:tokenization-example3} in
Appendix \ref{sec:tokenization-examples}.

%%%%%%%%%%%%%%%%%%%%%
\section{Translation Experiments}
\label{sec:translation-exp}
%%%%%%%%%%%%%%%%%%%%%

\footnotetext[5]{This vocabulary does not balance languages because
  the model is not precisely multilingual.}

%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental Settings}

We evaluated the translation quality as follows:

\paragraph{Tokenizer/vocabulary}

From the tokenizers/vocabularies used in Section
\ref{sec:tokenization-exp}, we selected all SentencePiece vocabularies
and mBART/XLM-R and mT5 as the baselines.

\paragraph{Translation Languages}

We selected the following eight out of 98 languages and trained a
multilingual translation model in all directions ($8 \times 7 = 56$
directions) for each vocabulary:

\begin{itemize}

\item \textbf{Latin Languages:} \\
English, Spanish, and Vietnamese:  We selected one European language
and one Asian language other than English.

\item \textbf{Single Languages:} \\
Japanese and Mandarin Chinese (Standard Beijing):  Although their
characters have the same origin, they use different glyphs (i.e.,
different character codes), in most cases.

\item \textbf{Other Languages:} \\
Modern Standard Arabic, Hindi, and Russian:
These are the other script types of the above languages.

\end{itemize}

\paragraph{Parallel Corpus}

We sampled 1 million sentences for each language pair from the
NLLB-200 corpus as the parallel corpus to train the translation
models.  We sampled sentences independently for each language pair.
Therefore, the importances of the languages are the same in this
experiment.

\paragraph{Translation Models}

We used the Transformer big models
\cite{DBLP:journals/corr/VaswaniSPUJGKP17} (1,024 embedding and 4,096
FFN dimensions, six layers for the encoder and decoder) implemented by
FairSeq \cite{ott-etal-2019-fairseq}, and learned multilingual models
in 56 ($8 \times 7$) directions.  Like the M2M-100 model
\cite{https://doi.org/10.48550/arxiv.2010.11125}, the multilingual
models were trained while we supplied language tags (e.g.,
`\texttt{\_\_en\_\_}' for English) at the head of the source and target
sentences.

\paragraph{Hyperparameters}

The details of the hyperparameters are shown in Appendix
\ref{sec:hyper-parameters}.

\paragraph{Evaluation}

We evaluated the translation quality using the average scores of 56
directions of ChrF++ \cite{popovic-2017-chrf} and COMET
\cite{rei-etal-2022-comet} (using the \texttt{wmt22-comet-da} model)
implemented in SacreBLEU \cite{post-2018-call}.  For the statistical
test, we used binomial testing with 56 trials, in which a trial
indicated a direction ($p < 0.05$).

%%%%%%%%%%%%%%%%%%%%%
\subsection{Results}

%%%%%%%%%%%%%%
\begin{table*}
\begin{center}
{\small
\begin{tabular}{l|lccccrr}
\toprule

\multicolumn{2}{c}{}
& Vocab.
& Byte
& Additional
& Lang.
& \multicolumn{2}{c}{Avg. score} \\

\multicolumn{2}{c}{Tokenizer/vocabulary}
& size
& fallback
& characters
& balance
& ChrF++ & COMET \\
\midrule

\multicolumn{2}{l}{Baselines} & \\
\cmidrule{2-8}

& XLM-R/mBART
& 250K & & & Corpus
& \textbf{41.13} & \textbf{.8237} \\

& mT5
& 250K & $\surd$ & & Corpus
& 40.44 & .8176 \\
\midrule

\multicolumn{2}{l}{SentencePiece/Unigram} & \\
\cmidrule{2-8}

& 250K\_C+B
& 250K & $\surd$ & 0 & Corpus
& \underline{40.93} & .8211 \\

& 250K\_S
& 250K & & 0 & Script
& 40.72 & .8167 \\

& 250K\_S+B
& 250K & $\surd$ & 0 & Script
& \underline{40.93} & \underline{.8212} \\

& 250K\_S+B+C52K
& 250K & $\surd$ & 52K & Script
& 40.89 & .8208 \\

& 64K\_S+B
& 64K & $\surd$ & 0 & Script
& 40.21 & .8139 \\

& 100K\_S+B+C52K
& 100K & $\surd$ & 52K & Script
& 40.09 & .8127 \\
\bottomrule
\end{tabular}
}
\end{center}
\caption{Translation quality for each tokenizer/vocabulary.  The
  tokenizer/vocabulary names of SentencePiece consisted of the
  vocabulary size, language balance (`C' is Corpus, and `S' is Script
  Type), byte fallback (B), and additional characters (C52K).  The
  bold scores indicate the highest score, and the underlined scores
  indicate the second-best scores.}
\label{tbl:vocab-trans-quality}
\end{table*}
%%%%%%%%%%%%%%

Table \ref{tbl:vocab-trans-quality} shows the translation quality for
each vocabulary.  Among all vocabularies, mBART/XLM-R achieved the
highest scores.  This is because it contained (not zero, but) very few
OOV tokens and the number of tokens was low.

Next, we focused on the results of our SentencePiece unigram models.
Regarding the vocabulary size, the translation qualities of the 250K
vocabularies were better than those of the 64k (or 100K) sizes.  For
example, the ChrF++ and COMET scores of \texttt{250K\_S+B} were higher
than those of \texttt{64K\_S+B} ($p = 1.6 \times 10^{-15}$), and the
scores of \texttt{250K\_S+B+C52K} were higher than those of
\texttt{100K\_S+B+C52K} ($p = 5.6 \times 10^{-17}$).

The translation quality with byte fallback was significantly higher
than that without byte fallback when comparing \texttt{250K\_S} and
\texttt{250K\_S+B} ($p = 2.5 \times 10^{-5}$), even though the
difference was small.

Regarding additional characters, although we could not find a
significant difference between \texttt{250K\_S+B} and
\texttt{250K\_S+B+C52K}, the scores of \texttt{64K\_S+B} were
significantly higher than those of \texttt{100K\_S+B+C52K} ($p = 6.1
\times 10^{-4}$).
%
Additional characters were not effective.  We suppose that this was
because multi-character subwords reduced in the vocabulary or
characters that were not learned remained when we added 52K characters.

%%%%%%%%%%%%%%%%%%%%%
\subsection{When Different Vocabulary Sizes are Used 
  between the Encoder and Decoder}

In the preceding discussion, we assumed that a shared vocabulary was
used in the encoder and decoder.  However, the optimal vocabularies of
the encoder and decoder may not be the same because the encoder is
responsible for natural language understanding, and the decoder is
responsible for generation.
%
Therefore, in this subsection, we confirm the translation quality if
we change the vocabulary between the encoder and decoder.

Specifically, we performed a translation experiment by changing the
vocabulary sizes between the encoder and decoder.  We used
\texttt{250K\_S+B} and \texttt{64K\_S+B} (i.e., the language balance
was the script type, with byte fallback, and the vocabulary sizes were
250K and 64K).

%%%%%%%%%%%%%%
\begin{table}
\begin{center}
{\small
\begin{tabular}{cccc}
\toprule
\multicolumn{2}{c}{Vocab. size}
& & \\
Encoder & Decoder
& ChrF++ & COMET \\
\midrule

\multicolumn{2}{c}{250K}
& \textbf{40.93} & \textbf{.8212} \\

250K & 64K
& 40.73 & .8192 \\
64K & 250K 
& 40.82 & .8203 \\

\multicolumn{2}{c}{64K}
& 40.21 & .8139 \\
\bottomrule
\end{tabular}
}
\end{center}
\caption{Translation quality (average scores in the 56 directions)
  when changing the vocabulary sizes of the encoder and decoder.}
\label{tbl:different-vocab}
\end{table}
%%%%%%%%%%%%%%

Table \ref{tbl:different-vocab} shows the result.  Regardless of
whether we changed the vocabulary size of the encoder or decoder, the
scores were intermediate between those of 250K and 64K.  The shared
vocabulary of the encoder and decoder was suitable to achieve high
translation quality.

%%%%%%%%%%%%%%%%%%%%%
\section{Comparison with Conventional Vocabulary Studies}
\label{sec:vocab-studies}
%%%%%%%%%%%%%%%%%%%%%

There have been various vocabulary studies using multilingual neural
models.  The findings of these studies, in comparison with the results
of our study, can be summarized as follows:

\citet{arivazhagan2019massivelymultilingualneuralmachine} built
multilingual models covering 103 languages using various conditions.
They also investigated vocabularies for the models and reported the
following findings.


\begin{enumerate}\itemsep=0mm
\item
Translation quality is better when a large vocabulary is used.

\item
Changes to the language balance of the vocabulary using temperature
sampling do not significantly affect translation quality.
\end{enumerate}

\noindent
In our experiments, a large vocabulary resulted in better translation
quality.  In addition, the language balance was not observed to have a
significant effect on quality.

\citet{gowda-may-2020-finding} investigated the optimal vocabulary
size for multiple languages (using only single-directional translation
models).  They reported that the optimal vocabulary size depends on
the training corpus size for the translation models.  Namely, a large
vocabulary is better in high-resource languages and a small vocabulary
is preferable low-resource languages.  As described in Section
\ref{sec:translation-exp}, our experiments indicate that a large
vocabulary is better because we use 1,000,000 parallel sentences for
each direction, which is regarded as a high-resource condition.

\citet{zhang-etal-2022-robust} constructed multilingual vocabularies
for eight languages with different English ratios in the training
corpora, and investigated the impact on the translation quality.  In
addition to the findings of conventional studies, they investigated
the effects of byte fallback and showed that this feature does not
significantly affect the translation quality.
%
In our experiments, in addition to eliminating OOV tokens, byte
fallback was found to enhance the translation quality.  Therefore, we
consider it preferable to use byte fallback.

%%%%%%%%%%%%%%%%%%%%%
\section{Conclusions}
\label{sec:conclusions}
%%%%%%%%%%%%%%%%%%%%%

In this paper, we discussed multilingual vocabulary for neural machine
translation models.  Our findings are summarized as follows:

\begin{enumerate}
\item
Among all vocabularies, mBART/XLM-R was the best in the machine
translation task.  Although the tokenizer of mBART/XLM-R did not use
byte fallback, the number of OOV tokens was small and, consequently,
the translation quality became high.

Among the vocabularies of our SentencePiece models, the vocabularies
of 250K with byte fallback achieved high quality.

\item
Byte fallback was effective for eliminating OOV tokens, and the
translation quality was better than that without byte fallback.

\item
The vocabularies of the 250K size generated the smallest number of
tokens (the shortest length of token sequences).  These vocabularies
had the disadvantage that the number of model parameters
increased. However, translation quality was better than that for the
64K vocabulary.

\item
To tokenize multilingual sentences into a similar (close) number of
tokens, it was effective to control the training data size of each
language. It could be controlled using a temperature coefficient.

\item
Readability increased when the number of fallbacked bytes was low.
However, translation quality decreased when we increased character
coverage by adding characters into the vocabulary.
\end{enumerate}

\paragraph{Recommended Vocabulary/Tokenizer}
Based on the vocabulary of mBART/XLM-R, we recommend using a tokenizer
with byte fallback.  In future work, we will build multilingual
translation models using the multilingual vocabulary discussed in this
paper.

%%%%%%%%%%%%%%%%%%%%%
\section*{Limitations}

The results in this paper were a case study
because our experiments were not comprehensive.

%%%%%%%%%%%%%%%%%%%%%
\section*{Ethics Statement}

Our vocabularies were created automatically from corpora, and we did
not check the contents.  Therefore, they may contain inappropriate
words.

%%%%%%%%%%%%%%%%%%%%%
\makeatletter\ifacl@finalcopy
\section*{Acknowledgments}

Part of this work was conducted under the commissioned research
program `Research and Development of Advanced Multilingual Translation
Technology' in the `R\&D Project for Information and Communications
Technology (JPMI00316)' of the Ministry of Internal Affairs and
Communications (MIC), Japan.
\fi\makeatother

%%%%%%%%%%%%%%%%%%%%%
% Bibliography entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
% Custom bibliography entries only
%\bibliography{custom}

%%%%%%%%%%%%%%%%%%%%%
\appendix

\section{Language List in this Paper}
\label{sec:language-list}

Table \ref{tbl:language-list} shows the list of 98 languages used in
this paper, which is organized by script type.

%%%%%%%%%%%%%%
\begin{table*}
\begin{center}
{\small
\begin{tabular}{lcp{103mm}}
\toprule
Script type & \#Langs. & Languages \\
\midrule

Arabic
& 6
& \textbf{Modern Standard Arabic}, Southern Pashto, Western Persian,
Sindhi, Urdu, Uyghur
\\

Armenian & 1 & Armenian \\

Bengali
& 2
& Assamese, Bengali
\\

Cyrillic
& 9
& Belarusian, Bulgarian, Kazakh, Kyrgyz, Macedonian, Halh Mongolian, 
\textbf{Russian}, Serbian, Ukrainian
\\

Devanagari
& 4
& \textbf{Hindi}, Marathi, Nepali, Sanskrit
\\

Ge'ez    & 1 & Amharic \\
Georgian & 1 & Georgian \\
Greek    & 1 & Greek \\
Gujarati & 1 & Gujarati \\
Gurmukhi & 1 & Eastern Panjabi \\

Hebrew & 2 & Hebrew, Eastern Yiddish \\

Hungul   & 1 & Korean \\
Japanese & 1 & \textbf{Japanese} \\
Kannada  & 1 & Kannada \\
Khmer    & 1 & Khmer \\
Lao      & 1 & Lao \\

Latin 
& 55
& Afrikaans, Tosk Albanian, North Azerbaijani, Basque, Norwegian
Bokmål, Bosnian, Catalan, Haitian Creole, Croatian, Czech, Danish,
Dutch, \textbf{English}, Esperanto, Estonian, Finnish, French,
Scottish Gaelic, Galician, Ganda, German, Hausa, Hungarian, Icelandic,
Igbo, Indonesian, Irish, Italian, Javanese, Northern Kurdish, Standard
Latvian, Lingala, Lithuanian, Plateau Malagasy, Standard Malay, West
Central Oromo, Polish, Portuguese, Romanian, Slovak, Slovenian,
Somali, \textbf{Spanish}, Sundanese, Swahili, Swedish, Tagalog /
Filipino, Tswana, Turkish, Northern Uzbek, \textbf{Vietnamese}, Welsh,
Wolof, Xhosa, Zulu
\\

Malayalam & 1 & Malayalam \\
Myanmar   & 1 & Burmese \\
Odia      & 1 & Odia \\
Simplified Chinese & 1 & \textbf{Mandarin Chinese (Standard Beijing)} \\
Sinhala   & 1 & Sinhala \\
Tamil     & 1 & Tamil \\
Telugu    & 1 & Telugu \\
Thai      & 1 & Thai \\
Traditional Chinese & 1 & Mandarin Chinese (Taiwanese) \\
\midrule

Total & 98 & \\
\bottomrule
\end{tabular}
}
\end{center}
\caption{Script types and languages. \#Langs. indicates the number of
  languages.  The languages in bold were used in the translation
  experiments.}

\label{tbl:language-list}
\end{table*}
%%%%%%%%%%%%%%

\section{Hyperparameters for Translation Experiments}
\label{sec:hyper-parameters}

Table \ref{tbl:hyper-parameters} shows the list of hyperparameters
used in the experiments in Section \ref{sec:translation-exp}.

%%%%%%%%%%%%%%
\begin{table*}
\begin{center}
{\small
\begin{tabular}{lll}
\toprule
Type & Name & Setting \\
\midrule

Model
& Architecture & Transformer big\\
& Embedding dimension & 1,024 \\
& FFN inner dimension & 4,096 \\
\midrule

Training
& Dropout & 0.3 \\
& Loss function & Label smoothed cross-entropy \\
& Label smoothing & $\epsilon = 0.1$ \\
& Optimizer & Adam ($\beta_{1} = 0.9$, $\beta_{2} = 0.98$) \\
& Learning rate & 5e-4 \\
& LR scheduler & Inverse square root \\
& Warm-up steps & 4,000 \\
& Global batch size & Roughly 128,000 tokens \\
& Early Stopping & No-update 9 epochs \\
\midrule

Test
& Beam width & 10 \\
\bottomrule
\end{tabular}
}
\end{center}
\caption{Hyperparameters for the translation experiments.}
\label{tbl:hyper-parameters}
\end{table*}
%%%%%%%%%%%%%%

\section{Tokenization Examples}
\label{sec:tokenization-examples}

Tables \ref{tbl:tokenization-example1} to
\ref{tbl:tokenization-example3} show tokenization examples, in which
the same sentences (or translations) obtained from the Flores+ dev set
were tokenized by each tokenizer.  Depending on the tokenizers, the
number of tokens vary significantly in a language, and each tokenizer
has strong and weak languages.  Among the tokenizers,
\texttt{mBART/XLM-R} and \texttt{250K\_S+B} tokenized the sentences
into fewer tokens on average.

%%%%%%%%%%%%%%
\begin{table*}
\begin{center}
\includegraphics[clip,height=190mm]{sample3-1.eps}
\end{center}
\caption{Tokenization examples obtained from the dev set in Flores+
  (1/3). The `$\sqcup$' and
  `\includegraphics[clip,width=10pt]{underbar.bmp}' symbols indicate
  the token delimiter and space character of SentencePiece,
  respectively.}
\label{tbl:tokenization-example1}
\end{table*}
%%%%%%%%%%%%%%

%%%%%%%%%%%%%%
\begin{table*}
\begin{center}
\includegraphics[clip,height=190mm]{sample3-2.eps}
\end{center}
\caption{Tokenization examples obtained from the dev set in Flores+
  (2/3). The `$\sqcup$' and
  `\includegraphics[clip,width=10pt]{underbar.bmp}' symbols indicate
  the token delimiter and space character of SentencePiece,
  respectively.}
\label{tbl:tokenization-example2}
\end{table*}
%%%%%%%%%%%%%%

%%%%%%%%%%%%%%
\begin{table*}
\begin{center}
\includegraphics[clip,height=190mm]{sample3-3.eps}
\end{center}
\caption{Tokenization examples obtained from the dev set in Flores+
  (3/3). The `$\sqcup$' and
  `\includegraphics[clip,width=10pt]{underbar.bmp}' symbols indicate
  the token delimiter and space character of SentencePiece,
  respectively.}
\label{tbl:tokenization-example3}
\end{table*}
%%%%%%%%%%%%%%

\end{document}

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: t
%%% tex-command: "platex"
%%% tex-dvi-print-command: "dvipdfmx -d 3 -f ptex-ipaex.map"
%%% End: 
