\section{Approach}

In this section, we will describe our reinforced learning and generation of adversarial examples (Figure \ref{fig:model_overview}) in detail. 
Overall, the victim model is a part of the \textbf{environment} (denoted as $Env$), which yields rewards indicating overall degradation based on modified inputs.
A reinforced \textbf{agent} learns to modify every source position from left to right sequentially. 
Meanwhile, a \textbf{discriminator} in $Env$ provides every-step survival signals by determining whether SRC is ill-perturbed.

\subsection{Environment}
We encapsulate the victim translation model with a discriminative reward process as an $Env$ for a reinforced agent to interact.

\subsubsection{Environment State}
The state of the $Env$ is described as $s_t = (SRC, t)$, where $SRC=[src_0, src1,...,src_N]$ are $N$ sequences processed by victim model's vocabulary and tokenization. 
Each sequence $src_i=[x_1, x_2, ..., x_n]$ is concatenated with $BOS, EOS$, which indicate the begin and end of the sequence, then padded to same length. 
Time step $t \in[1,n]$ also indicates the token position to be perturbed by the agent. 
$Env$ will consecutively loop for all token positions and update $s_t$ based on the agent's modification.
$Env$ also yields reward signals until the end or intermediately terminated. 
That is, \textit{all} sequences in $SRC$ are determined by $D$ as ill-perturbed during the reward process. 
Once the $Env$ terminates, it finishes the current episode and reset its state with a new batch of sequences as $SRC$.

\subsubsection{Reward Process with Discriminator}  \label{subsec:Reward_process}
The reward process is \textit{only} used during training. It consists of a survival reward $r_{\text{s}}$ on every step and a final degradation $r_{\text{d}}$ concerning an overall metric if the agent survives till the end. Overall, we have:
\begin{align}
\centering
r_t=
\begin{cases}
-1, \; \text{terminated} \\
\frac{1}{N}{\sum_N a \cdot r_{\text{s}}}, \;\text{survive}\And t\in [1,n) \\
\frac{1}{N}{\sum_N (a \cdot r_{\text{s}}+ b \cdot r_{\text{d}})}, \;\text{survive}\And t=n 
\label{eq:rewards}
\end{cases}
\end{align}
where $a,b$ are hyper parameters that keeps the overall $r_{\text{s}}$ and $r_{\text{d}}$ within similar magnitude.

Instead of direct optimization of the constrained adversarial loss in Eq.\ref{eq:adversarial_loss_continuous}, we model discriminator $D$'s output as survival rewards similar to that in gaming \citep{mnih2015human}.
That is, the agent must survive for its goal by also fooling $D$, which attempts to terminate ill-perturbed modifications.
We define an ill-perturbed source by determining whether it still matches the original target $tgt$. 

\paragraph{Discriminator} As it is shown in Figure \ref{fig:model_overview}(b), discriminator $D$ consists of bi-directional GRU encoders for both source and target sequence. Their corresponding representation is averaged and concatenated before passed to a feedforward layer with dropout. Finally, the output distribution is calculated by a softmax layer.
Once $D$ determines the pair as positive, its corresponding possibility is regarded as the reward, otherwise 0:
\begin{align}
\centering
r_{\text{s}}=
\begin{cases}
P(\text{positive}|(src',tgt); \theta_d), & \text{positive} \\
0, & \text{otherwise}\\
\end{cases}
\label{eq:r_degrade}
\end{align}

As long as the environment survives, it yields averaged reward among samples from $SRC$ (Eq.\ref{eq:rewards}) to mitigate rewards' fluctuation that destabilize training. 

\paragraph{Discriminator Training}
Similar to GAN training, the environment's $D$ must update as the agent updates. During its training, the agent's parameter is freezed to provide training samples. 
% We treat original pair $(src, tgt)$ as positive sample, while $(src', tgt)$ as negative.
For every $D$'s training epoch, we randomly choose half of the batch and perturb its source using the current agent as negative samples. 
During $D$'s updates, we randomly generate a new batch of pairs from parallel data likewise to test its accuracy. 
$D$ is updated at most $step_D$ epochs, or until its test accuracy reaches ${acc\_bound}$.

$Env$ \textit{only}\footnote{It is commonly accepted that frequent negative rewards result in agents' tendency to regard zero-reward as optimum and fail exploration, which further leads to training failure.} yields -1
as overall terminal rewards when all sequences in $SRC$ are \textit{intermediately} terminated. For samples classified as negative during survival, their follow-up rewards and actions are masked as 0. 
If the agent survives until the end, $Env$ yields additional averaged $r_{\text{d}}$ as final rewards for an episode. 
We follow \newcite{michel2019evaluation} to adopt relative degradation:
\begin{align}
\centering
    r_{\text{d}}=\frac{\text{score}(y,refs)-\text{score}(y',refs)}{\text{score}(y,refs)}
    \label{eq:RD}
\end{align}
where $y$ and $y'$ denote original and perturbed output, $refs$ are references, and $score$ is a translation metric. If $score(y,refs)$ is zero, we return zero as $ r_{\text{d}}$. 
To calculate $\text{score}$ we \textit{retokenize} perturbed $SRC$ by victim models vocabulary and tokenizer before translation.

\subsection{Agent}
As it is shown in Figure \ref{fig:model_overview} (c), the agent's actor and critic share the same input layers and encoder, but later processed by individual feedforward layers and output layers. Actor takes in $SRC$ and current token with its surrounding $(x_{t-1}, x_t, x_{t+1})$, then yields a binary distribution to determine whether to attack a token on step $t$, while critic emits a value $V(s_t)$ for every state.
Once the actor decides to perturb a specific token, this token will be replaced by another token in its candidate set.

\paragraph{Candidate Set}
We collect at most $K$ candidates for each token in the victim's vocabulary within a distance of $\epsilon$. $\epsilon$ is the averaged Euclidean distance of $K$-nearest embedding for all tokens in victim vocabulary. 
We note that there shall always be candidates for a token in test scenarios that are beyond victim's vocabulary, for those without a nearby candidate, we assign UNK as its candidate.
Once the agent chooses to replace a token with UNK, we follow \newcite{michel2019evaluation} to present a valid token that is also UNK to the victim's vocabulary.

\paragraph{Agent Training}
The agent is trained by algorithm in appendix A.
% , where we apply asynchronous version of actor critic to ensure stability \citep{mnih2015human}. That is, the agent explore and collect gradients based on local set of parameters to update global parameters, then only when environment finishes an episode, they are synchronized with the global parameters. 
Since the agent is required to explore with stochastic policy during training, it will first sample based on its actor's output distribution on whether to perturb the current position, then randomly choose among its candidates.
The agent and discriminator take turns to update. 
We assume the training is converged when test accuracy for $D$ does not reach over a certain value within certain continuous learning rounds of agent and discriminator.
\paragraph{Agent Inference}
To generate adversarial examples, the agent will take in source sequences and perturb on each position based on the actor's output from left to right, then choose the nearest candidate. 
As the agent's critic learns to estimate expected future rewards for a step, only when it yields positive value will agent perturb, otherwise it indicates an undesirable perturbation; thus, the agent is muted. 
