\appendix
\section{Training Details for Agent}

We adopt commonly accepted translation metric BLEU as $score$ in Eq.\ref{eq:r_degrade}.
We use 50 sequence pairs per batch both in environment initialization and training of discriminator and agent.
It is essential to train on batches of sequences to stabilize reinforced training.
Furthermore, note that $D$ can be too powerful during the early training stage compared to the agent's actor that it can quickly terminate an exploration. Therefore, we must train on batches and determine an overall terminal signal as aforementioned to ensure early exploration.
The $step_D$ and $step_A$ are set as $80$ \footnote{ Three times the average convergence episodes to train a discriminator with initial agent by the given batch size.} and $120$. $acc\_bound$ for discriminator training is set to $0.85$. The $a$ and $b$ in Eq.\ref{eq:rewards} are set to $0.5$ and $10$. 
The dimension of feedforward layers in the agent's actor-critic and discriminator are all $256$.
We initialize the embedding of both agent and discriminator by the victim's embedding.

For reinforcement learning, we adopt asynchronous learning with an additional global agent with an additional set of parameter $\theta^{\Omega}$, we set discount factor $\gamma$ to $0.99$, $\alpha$ and $\beta$ in Eq.\ref{eq:overall_loss} to $0.5$ and $0.05$ respectively. 
As for the stop criterion, we set $patience\_round$ to $15$ with convergence boundary for $acc_D$ to $0.52$.
We adopt Adafactor\citep{shazeer2018adafactor} for training, which is a memory-efficient Adam. The learning rate for agent's optimizer is initiated as $0.001$ and scheduled by rsqrt with 100 steps of warmup. The $K$ for the candidate set is 12.

Our agent takes around 30 hours to converge on a single Nvidia 1080ti. 
\label{sec:Alg_train}
\begin{algorithm}[!ht]
  \SetAlgoLined
  \KwResult{A learned global agent $\pi_{\theta^{\Omega}}$}
  \text{Assume global agent as $\pi_{\theta^{\Omega}}$ with parameter $\theta^{\Omega}$ } \\
  \text{Assume agent as $\pi_\theta$ with parameter set $\theta$} \\
  initialize: $Env$ with $D$, $\theta^{\Omega}$, $\theta$ \;
  \While{\text{not Stop Criterion}}{
    \For{$step_D$}{
     train $D$ with current agent $\pi_\theta$ \;
     if {$acc_D > acc\_bound$ }
        break\;
    }
    test current $D$'s accuracy $acc_D$ for stop criterion\;
    \For{$step_A$}{ 
      initialize $Env$ state $s_0$\; 
      synchronize $\pi_\theta$ with $\pi_{\theta^{\Omega}}$ \;
      $t=t_{start}$ \;
      \While{\text{$s_t$ survive and $t-t_{start}<t_{max}$}}
      {get $out_t^{\text{actor}}, V_t=\pi_\theta(s_t)$ \;
       compute entropy $H(out_t^{\text{actor}})$ \;
       sample $a_t$ based on $out_t^{\text{actor}}$ \;
       perform $a_t$ and receive $r_t$ and $s_{t+1}$ \;
       $t\leftarrow t+1$ \;
      }
      $R=\begin{cases}
        0   & \text{for terminal $s_t$} \\
        V(s_t)  &\text{for non-terminal $s_t$} \\
        \end{cases}$  \\
      \For{$i\in\{t-1,...,t_{start}\}$ }
      {  $R\leftarrow \gamma R + r_i $ \;
         accumulate $L_t^v(\theta)$ \;
         accumulate $L_t^{\pi}(\theta)$ \;
      }
      compute overall loss $L(\theta)$ \;
      perform asynchronous updates on $\theta^{\Omega}$ with gradient $\frac{\partial L(\theta)}{\partial \theta}$ \;
    }
   }
  \caption{Reinforced training for agent}
  \label{alg:train_overview}
\end{algorithm}
Note that higher $acc\_bound$ and lower convergence boundary for $D$ indicates higher semantic constraints, which will increase training time. 

\section{Search-based Attack}
Search-based adversarial generation is currently widely applied in various robustness machine translation system. 
We generally follow the strategy of \newcite{ebrahimi2018adversarial,michel2019evaluation}  which is applicable for both RNN-search and Transformer. More specifically, the $L_{adv}$ in Eq.\ref{eq:adversarial_loss_discrete} is derived as:

\begin{align}
\centering
    &\underset{1 \leq i \leq n, emb_i' \in vocab}{\mathrm{argmax}}|emb'-emb_i| \nabla _{emb_i}L_{adv}, \\
    &L_{adv}(X',Y) = \sum^{|y|}_{t=1} log(1-P(y_t|X',y_1...y_{t-1}))\nonumber
\end{align}
where each $P(y_t|X)$ is calculated by Eq.\ref{eq:nmt_output} given a corresponding target.
For every source sequence, a small ratio of positions is sampled for search. Then we greedy search\footnote{\newcite{ebrahimi2018adversarial} suggest that greedy search is a good enough \textbf{approximation}.} by the corresponding loss upon those positions with given candidates.
For better comparison, we adopt the candidate set used in our model instead of naive KNN candidates. Both baseline and our model share the same UNK generation for presentation. We use homophone replacement for Chinese, and strategy by \newcite{michel2019evaluation} for English.  


