\section{Analysis} \label{analysis}

% \begin{figure}
%     \centering
%     \includegraphics[width=0.5\textwidth]{action_entropy.pdf}
%     \caption{Action entropy during first 13000 training episode}
%     \label{fig:action_entropy}
% \end{figure}

% \begin{figure}
%     \centering
%     \includegraphics[width=0.5\textwidth]{discriminator_acc.pdf}
%     \caption{Discriminator accuracy during training.}
%     \label{fig:discriminator_acc}
% \end{figure}

\subsection{Efficiency}

As it is shown in Figure \ref{fig:time_cost}, given the same amount of memory cost, our method is significantly more efficient compared to the search paradigm. Gradient computation concerning every modified source sequence can cost considerably in time or space for a state-of-the-art system, which could be even worse for systems with recurrent units. 
When it comes to mass production of adversarial examples for a victim translation system, our method can also generate by given only monolingual inputs. In contrast, search methods must be provided the same amount of well-informed targets.

\begin{figure}[!h]
    \centering
    \includegraphics[width=0.45\textwidth]{pictures/timecost.pdf}
    \caption{Time consumption of different methods: we limit memory usage to 2.5G on single Nvidia 1080, and generate adversarial examples for the same $800$ inputs in Zh$\rightarrow$En MT with different methods, our method significantly outclasses the state-of-the-art search paradigm (GS).}
    \label{fig:time_cost}
\end{figure}

\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth]{pictures/pattern.pdf}
    \caption{Attack preferences of different paradigms targeting Zh$\rightarrow$En Transformer-BPE model. All share a similar modification rate. Our agent shows a significant preference for some POS (e.g., Ni, Nh, Nz, I), which are commonly regarded as hard-to-translate phrases among industrial implementations, while some (e.g., K) are less noticed.
Preference among different choices.}
    \label{fig:attack_pattern}
\end{figure*}

\subsection{Attack Patterns}
NMT systems may have different robustness over different parts of the inputs, thus some researchers implement input preprocessing targeting certain empirically weak parts, e.g., named entities\citep{li-etal-2018-named}.
Since the agent's policy is to attack without handcrafted error features, we can further investigate vulnerability by its attack preferences of different parts of speech.
We choose Chinese, for example, and adopt LTP POS tagger\footnote{https://github.com/HIT-SCIR/ltp} to label NIST test sets, then check the modification rate for each POS. To ensure the reliability of our analysis, we run three rounds of experiments on both baselines and our agent with \textit{similar modification rate} targeting state-of-the-art Transformer with BPE, and collect overall results. We also present random synthetic noise injection \citep{karpukhin2019training} (RSNI), which is not intended for any preference as an additional baseline. 

As it is shown in Figure \ref{fig:attack_pattern}, our reinforced paradigm shows distinct preference upon certain POS tags, indicating pitfalls of a victim translation system. 
At the same time, RSNI distributed almost evenly upon different POS tags. 
Though the search paradigm (GS) does expose some types of pitfall, our method can further expose those omitted by the search.
Note that unlike existing work relying on feature engineering to indicate errors, we have no such features implemented for an agent. However, our agent can still spot error patterns by favoring some of the POS, such as Ni (organization name), Nh (person name), Nl (location name), M (numbers), which are commonly accepted as hard-to-translate parts. Moreover, the agent also tends to favor K (suffix) more, which is less noticed. 

% \subsection{Attack by UNK}
% As it is commonly acccepted that UNK label can greatly affects a NMT system, users are very likely to break a MT system by introducing UNK due to typos or misspelling. Thus we want to inspect what POS are most likely to be affected by UNK replacement.
% Therefore, we further launched the attacks where we skip all the replacement for an valid token to UNK. As expected, 
% \ZW{both less modification change search more dependent on UNK replacement} 

\subsection{Attack Generalization}
We additionally test agents by attacking different model architecture from the one that it's trained. 
As it is shown in Table \ref{tab:dis_attack}, we perturb the inputs by agents trained to attack a different architecture, then test for degradation.
The results show that our agent trained by targeting Transformer architecture can still achieve degradation on RNN-search, and vice-versa. 
% We also notice that for En$\rightarrow$Fr and En$\rightarrow$De, the agent trained by targeting Transformer-BPE is substantially stronger for RNN-search-BPE attack , while Zh-En is the opposite.

\begin{table}[]
    \centering
    {
    \resizebox{0.45\textwidth}{!}{
        \begin{tabular}{lll}
\hline
                                                  & \multicolumn{1}{l|}{Attack by}     & BLEU$(\Delta)$             \\ \cline{2-3} 
                                                  & \multicolumn{2}{c}{Zh-En MT02-06}                     \\ \hline
\multicolumn{1}{l|}{\multirow{3}{*}{RNN-search-BPE}}  & \multicolumn{1}{c|}{-}             & 40.90            \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-RNN}     & 31.58(-9.32)     \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-TF}      & 32.14(-8.76)     \\ \hline
\multicolumn{1}{l|}{\multirow{3}{*}{Transformer-BPE}} & \multicolumn{1}{c|}{-}             & 44.06            \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-TF}      & 35.48(-8.58)     \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-RNN}     & 33.14(-10.92)    \\ \hline
                                                  & \multicolumn{2}{c}{En-De Newstest13-16}               \\ \hline
\multicolumn{1}{l|}{\multirow{3}{*}{RNN-search-BPE}}  & \multicolumn{1}{c|}{-}             & 25.35            \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-RNN}     & 21.27(-4.08)     \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-TF}      & 17.18(-8.18)     \\ \hline
\multicolumn{1}{l|}{\multirow{3}{*}{Transformer-BPE}} & \multicolumn{1}{c|}{-}             & 29.05            \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-TF}      & 19.29(-9.76)     \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-RNN}     & 24.2(-4.85)      \\ \hline
                                                  & \multicolumn{2}{c}{En-Fr Newstest13-14+newsdiscuss15} \\ \hline
\multicolumn{1}{l|}{\multirow{3}{*}{RNN-search-BPE}}  & \multicolumn{1}{c|}{-}             & 32.60            \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-RNN}     & 22.3(-10.30)     \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-TF}      & 19.83(-14.87)    \\ \hline
\multicolumn{1}{l|}{\multirow{3}{*}{Transformer-BPE}} & \multicolumn{1}{c|}{-}             & 34.70            \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-TF}      & 21.33(-13.37)    \\
\multicolumn{1}{l|}{}                             & \multicolumn{1}{c|}{agent-RNN}     & 22.35(-10.25)    \\ \hline
\end{tabular}}
    \caption{Attacks targeting different architecture from the trained one. We note agent with the architecture that is trained with(e.g., agent-RNN stands for agent trained by targeting RNN-search).}
    \label{tab:dis_attack}
    }
\end{table}

\subsection{Tuning with Adversarial Examples}

Since the agent generates meaning-preserving adversarial examples efficiently, we can directly tune the original model with those samples. 
We choose Zh$\rightarrow$En Transformer-BPE, for example, and generate the same amount of adversarial examples given original training sources(1.3M pairs), then paired with initial targets.
We mix the augmented pairs with original pairs for a \textit{direct} tuning. 
We test the tuned model on original test data and noisy test data generated by the attacking agent.
We additionally test on IWSLT11-17 Zh$\rightarrow$En test data, which is not used for training or tuning, to verified robustness improvement. 
As Table \ref{tab:tuning} shows, our agent can achieve substantial improvement(+4.83) on noisy tests with only minor loss on clean tests(-0.46).
The improvement on the IWSLT test also indicates the adversarial tuning contributes to not only defending the agent's attack, but also overall robustness.

\begin{table}[]
    \centering
    \resizebox{0.45\textwidth}{!}{
    \begin{tabular}{ll|l|l}
\hline
                                     & Clean test & Noisy test & IWSLT11-17 \\ \hline
\multicolumn{1}{l|}{Transformer-BPE} & 44.06     & 35.48            & 11.27      \\ \hline
\multicolumn{1}{l|}{Tuned}           & 43.60(-0.46)     & 40.31(+4.83)            & 11.73(+0.46)      \\ \hline
\end{tabular}}
    \caption{Tuning Zh$\rightarrow$En Transformer-BPE model with adversarial examples. We generate adversarial examples for every training sources for tuning, achieving overall improvements for noisy tests.}
    \label{tab:tuning}
\end{table}


\begin{table}[!t]
    \centering
    \resizebox{0.47\textwidth}{!}{
        \begin{tabular}{c|l}
        \hline
        \textbf{in} & \begin{CJK}{UTF8}{gkai}钱其琛同突尼斯外长会谈。\end{CJK} \\
        \hline
        \textbf{perturbed in} & \begin{CJK}{UTF8}{gkai}钱其琛同突尼斯外长会谈\textbf{\textit{-}}\end{CJK} \\
        \hline
        \textbf{out} & Chinese, Tunisian minsters hold talks. \\
        \hline
        \textbf{perturbed out} & qian qichen holds talks with Tunisian foreign minister. \\
        \hline
        \hline
        \textbf{in} & \begin{CJK}{UTF8}{gkai}中巴及城巴车辆在南区通宵停泊\end{CJK} \\
        \hline
        \textbf{perturbed in} &\begin{CJK}{UTF8}{gkai}中巴及城巴车辆在南区通宵停\textbf{\textit{车}}\end{CJK} \\
        \hline
        \textbf{out} & overnight parking of cmb and city bus \\
        \hline
        \textbf{perturbed out} & overnight parking of cmb and city bus in southern district \\
        \hline
        \end{tabular}
    }
    \caption{Example of minor perturbed samples that improves machine translation for Zh$\rightarrow$En Transformer-BPE model. The ``\begin{CJK}{UTF8}{gkai}。\end{CJK}'' in first sample is modified to ``-'', then model yields the omitted ``\begin{CJK}{UTF8}{gkai}钱其琛\end{CJK} (qian qi chen)''. The ``\begin{CJK}{UTF8}{gkai}停泊\end{CJK}'' in second sample is modified to ``\begin{CJK}{UTF8}{gkai}停车\end{CJK}'', where they both mean ``parking'', then comes the omitted ``in southern district'' for ``\begin{CJK}{UTF8}{gkai}在南区\end{CJK}''. }
    \label{tab:reinforced_examples}
\end{table}

\subsection{Reinforced Examples for Machine translation}
We additionally switched the episodic rewards in the environment, then ignored all modifications that induce UNK tokens to train an agent, hoping to generate minor perturbed samples that can improve the translation metric. 
Though we failed to achieve overall improvements, we do succeed for quite a portion of samples, as shown in Table \ref{tab:reinforced_examples}. 
Similar to adversarial examples, we call them \textit{reinforced examples}. 
Such improvement is different from adversarial training that tunes model for defense or strict text correction before the test phase. 
Reinforced examples are still noisy and can be directly applied for a test without any model updates to achieve improvements, which to our best knowledge is less investigated by researchers.
Since we discovered that not all perturbed inputs are harmful, such an issue can be a good hint and alternative for better adversarial defense in NLP and should be further considered.