\section{Basic Models}

\textbf{The ORL model} aims to extract \textit{opinion-holder-target} structures from text by identifying the segments of these opinion arguments. The task can be modeled as a sequence labeling problem. We adopt the $\{BMESO\}$ encoding schema to assign a tag for each word \cite{zhang2019enhancing}.
%convert opinion arguments into a sequence of boundary tags for each word. 
Following \citet{marasovic2017srl4orl} and \citet{zhang2019enhancing}, we focus on recognizing the holders and the targets for the given opinion expression and exploit a deep BiLSTM-CRF-based model as our baseline. %since it works well over the sequence labeling tasks. 

The Figure \ref{fig:orl-parser}-(a) shows the architecture of our ORL baseline model, which is composed of three key components, i.e., the input layer, the BiLSTM-based encoder, and the CRF-based decoder. % todo zy: baseline model not in the right of figure
%The basic model includes three parts: the input layer, the deep BiLSTM encoder and the CRF decoder.
Given the input sentence $S=w_1, w_2, ..., w_n$ and the opinion expression segment $E = w_s, w_{s+1}, ..., w_e (1\leq s\leq e\leq n)$, the input vector consists of the word embeddings and the expression-indicator embeddings as the following equation shows:
\begin{equation}\label{eq:input}
\begin{split}
    \mathbf{x}_{i} &= \mathbf{e}^{word}_{w_i} \oplus \mathbf{e}^{exp}_{0/1} 
\end{split}
\end{equation} 
where $\mathbf{e}^{word}_{w_i}$ is the embedding of word $w_i$, and the expression-indicator embedding is $\mathbf{e}^{exp}_0$ for non-expression words  and $\mathbf{e}^{exp}_1$ for words inside the opinion expression (i.e., $s \leq i \leq e$).  % todo zy 这段0-1表示，没看明白
At the encoder layer, we apply three stacking layers of BiLSTM to fully encode the sentence and obtain the expression-specific representations at word level. The CRF-based decoder at the output layer delivers the globally optimal sequence tags.
% For Input layer, word embedding concat with expressions indice embedding. Formally, the input representation of the $w_i$.
 %a parser ﬁrst scores all word pairs about how possible they hold valid dependency relations. means how possible all word pairs hold valid dependency relations

 %getting the high-level word representations
 % and obtain representations with syntactic information at word level, i.e, the lstm hidden states of BiLSTM layer.   , obtaining a probability matrix that can be viewed as a fully connected graph with edge weight after a softmax operation. 
 %  (The probability matrix can be viewed as a fully connected graph with edge weight).
 %which obtain a $n*n$ probability matrix after a softmax operation (also can be viewed as a fully connected graph with edge weight). (batch equatoin???)
\textbf{The Biaffine parser} is the state-of-the-art dependency parser proposed by \citet{dozat2016deep}, as shown in Figure \ref{fig:orl-parser}-(b). %tody zy: 我们是缺了一张图吗？怎么感觉fig1没有呢
The parser contains a multi-layer BiLSTM layer for encoding the input sentence, followed by a biaffine transformation layer for computing the probabilities of all word pairs. Then it searches for the highest-scoring and well-formed tree via the maximum spanning tree (MST) algorithm. % over the edge-weighted graph.  after the dimension-reducing MLP layers

%(can be viewed as a weighted fully directed graph)
%label which word is its head word for each word, indicates the dependency relation between word pairs of sentence and 
The three cascaded layers, i.e., the BiLSTM-based encoder, the biaffine scorer, and the MST decoder, represent syntactic information at different levels. The encoder extracts the neural features from the input sentence and outputs hidden states (HDN), which can be regarded as \textit{implicit} information. The 1-best output parse tree, on the other hand, conveys \textit{explicit} syntactic structures. The biaffine scorer gives a probability matrix for all possible dependency arcs (also can be viewed as an edge-weighted directed graph), which represents richer \textit{explicit} syntactic information than the 1-best parse tree. %called as syntax representation (SYNREP) 

% \textbf{The Syntax-Aware Models} from previous works are investigated as our baselines. The \textbf{head embedding} approach is the most intuitive way to represent syntactic information by using individual dependency heads. Here we use the embedding of the head word of $w_i$ as the extra input of $w_i$.
% % The \textbf{TPF} (tree position feature) approach extracts the relative positions of $w_i$ and $w_e$ to their common ancestor word in the dependency tree and map such features into the embeddings as extra representations of the word (cf. \citet{xia2019syntax} and \citet{yang2016position} for details), where $w_e$ is the head word of the opinion expression.
% The \textbf{TreeLSTM} approach stacks the TreeLSTM encoder over the regular LSTM encoder to represent the parse tree, and feed the outputs into the CRF-based decoder.
% % The \textbf{GCN} approach\cite{marcheggiani2017encoding} use multi-layer GCN encoder to extrect syntactic information from the parsed tree and use the output of it as the input of the CRF decoder.
% The \textbf{DepHidden} (dependency parser hidden states) approach extracts the hidden states of the parser encoder and use them as the extra input embeddings.

%, which will be introduced in 

% Previous works propose a various of approaches for encoding syntactic information to build better syntax-aware model. They tend to only pay attention on exploiting the parsed dependency tree or the SYNREP. 
% For exploiting the parse tree, on the on hand, the TreeLSTM/GCN encoder\cite{tai2015improved,marcheggiani2017encoding}, has commonly be used to extract syntactic information. On the other hand, some works investigate to exploit the part of the tree for attending to more important information, such as the shortest path tree (SDP)\cite{??}, tree position feature (TPF)\cite{???}. For syntax-aware representation, \cite{????} extract them from a well-trained parser and is fed into target model as extra inputs.

%which contains all latent dependency trees
% In this work, we would integrate syntactic information from two different levels. The one is the syntactic information of the highest-scoring tree, only one ("hard"). The other is relatively softer representations of syntax, i.e, the weighted fully directed graph ( also can be viewed as all latent trees) and the obscure lstm hidden states of the parser.

% \begin{equation}
% \bm{rep}^{syn}_i = \lambda\sum_{1 \leq j \leq N} \alpha_j  \bm{h}^j_i
% \end{equation}

%\subsection{Graph Convolutional Network}

