\section{Introduction}  \label{introduction}

% \SJ{main contribution:}
% \SJ{
% 1. free of bias induced by feature engineering
% 2. free of accessory data for constraint modeling
% 3. robust analysis over different model
% }

\noindent Neural machine translation (NMT) based on the encoder-decoder framework, such as RNN-Search \citep[RNNSearch]{Bahdanau2015Neural, Luong2015Effective}
or Transformer \citep[Transformer]{vaswani2017attention}, has achieved remarkable progress and become a de-facto in various machine translation applications. However, there are still pitfalls for a well-trained neural translation system, especially when applied to less decent real-world inputs compared to training data \citep{belinkov2017synthetic}. For example, typos may severely deteriorate system outputs (Table \ref{tab:brittle_nmt}). 
Moreover, recent studies show that a neural machine translation system can also be broken by noisy synthetic inputs \citep{belinkov2017synthetic,lee2018hallucinations}.
Due to the black-box nature of a neural system, it has been a challenge to fathom when and how the system tends to fail.

% bad case table
\begin{table}[!t]
    \centering
    \resizebox{0.45\textwidth}{!}{
        \begin{tabular}{c|l}
        \hline
        \textbf{in} & \begin{CJK}{UTF8}{gkai}耶路撒冷发生自杀\textit{\textbf{爆炸}} 事件 \end{CJK}  \\
        \hline
        \textbf{out} & suicide bombing in jerusalem \\
        \hline
        \textbf{in} & \begin{CJK}{UTF8}{gkai}耶路撒冷发生自杀\textit{\textbf{爆}}事件\end{CJK} \\
        \hline
        \textbf{out} & \textit{eastern} jerusalem \textit{explores a case of eastern europe} \\
        \hline
        \end{tabular}
    }
    \caption{Fragility of neural machine translation. A typo leaving out a Chinese character ``\begin{CJK}{UTF8}{gkai}炸\end{CJK}'' leads to significant errors (noted by italics) in English translation. Both ``\begin{CJK}{UTF8}{gkai}爆\end{CJK}'' and ``\begin{CJK}{UTF8}{gkai}爆炸\end{CJK}'' mean ``bombing'' in English.}
    \label{tab:brittle_nmt}
\end{table}

Intuitively, researchers seek to apprehend such failures by the analysis of handcrafted error indicating features~\citep{zhao2018addressing, karpukhin2019training}. This strategy is costly because it requires expert knowledge for both linguistics and the target neural architecture. Such features are also less applicable because some common errors in deep learning systems are hard to formulate, or very specific to certain architectures.

%% adversarial learning paradigm
Instead of designing error features, recent researchers adopt ideas from adversarial learning~\citep{goodfellow2014explaining} to generate adversarial examples %by given metrics
for mining pitfalls of NLP systems ~\citep{cheng2018seq2sick,ebrahimi2018adversarial,zhao2017generating}.
Adversarial examples are minor perturbed inputs that keep the semantic meaning, yet yield degraded outputs.
The generation of valid adversarial examples provides tools for error analysis that is interpretable for ordinary users, which can contribute to system maintenance.
Though it has achieved success concerning continuous input, e.g., images, there are following major issues for NLP tasks. 

First, it is non-trivial to generate valid discrete tokens for natural language, e.g., words or characters. 
\newcite{cheng2018seq2sick} follow \newcite{goodfellow2014explaining} to learn noised representation then sample tokens accordingly. However, there is no guaranteed correspondence between arbitrary representation and valid tokens. Therefore, it may generate tokens departing from learned representation, which undermines the generation.
%\SJ{bringing gap to final degradation}. 
\newcite{ebrahimi2018adversarial} turns to a search paradigm by a brute-force search for direct perturbations on the token level.
To lead the search, a gradient-based \textit{surrogate} loss must be designed upon every token modification by given target annotations. 
However, this paradigm is inefficient due to the formidable computation for gradients. 
Furthermore, surrogate losses defined upon each token by targets requires high-quality targets, and risks being invalidated by any perturbation that changes tokenization.

Another issue is to keep the semantics of original inputs. 
Different from the fact that minor noises on images do not change the semantics, sampling discrete tokens from arbitrary perturbed representation \citep{cheng2018seq2sick} may generate tokens with different semantics and lead to ill-perturbed samples (Table \ref{tab:unwanted_perturb}).
Searching for the perturbed input also requires a semantic constraint of the search space, for which handcrafted constraints are employed \citep{ebrahimi2018adversarial}.
Though constraints can also be introduced by multitask modeling with additional annotations \citep{zhao2017generating}, this is still not sufficient for tasks requiring strict semantic equivalence, such as machine translation. 

% unwanted perturb
\begin{table}[!t]
    \centering
    \resizebox{0.4\textwidth}{!}{
        \begin{tabular}{c|l}
        \hline
        \textbf{in} & Two man are playing on the street corner. \\
        \hline
        \textbf{perturbed in} & Two man are playing \textit{\textbf{frisbee in the park}}. \\
        \hline
        \textbf{out} & Zwei Männer spielen an einer Straßenecke. \\
        \hline
        \textbf{perturbed out} & Zwei Männer spielen frisbee im park. \\
        \hline
        \end{tabular}
    }
    \caption{Example of undesirable perturbation in adversarial examples for machine translation in  \citep{zhao2017generating}, though it yields very different output compare to the origin, it does not indicate system malfunction.}
    \label{tab:unwanted_perturb}
\end{table}

% \ZW{discrete generation & semantic constraints}
In this paper, we adopt a novel paradigm that generates more reasonable tokens and secures semantic constraints as much as possible. We summarize our contributions as the following:
\begin{itemize}

\item 
We introduce a reinforcement learning \citep[RL]{sutton2018reinforcement} paradigm with a discriminator as the terminal signal in its environment to further constrain semantics. This paradigm learns to apply discrete perturbations on the token level, aiming for direct translation metric degradation. 
Experiments show that our approach not only achieves semantically constrained adversarial examples but also leads to effective attacks for machine translation.

\item
Our paradigm can achieve the adversarial example generation with outclassed efficiency by only given source data.
Since our method is model-agnostic and free of handcrafted error feature targeting architectures, it is also viable among different machine translation models.

% \ZW{efficiency and attack pattern}
\item 
We also present some analysis upon the state-of-the-art Transformer based on its attack, showing our method's competence in system pitfall exposure. 
% indicating an architecture's robustness on certain inputs by an evaluation%
\end{itemize}