\section{The Syntax-Aware Approach}
\label{sec:proposed}

%In this work, we utilize dependency syntactic information  at different  processing  levels of the parser model, and apply different encode approaches to integrate syntactic information to ORL model.

%parser-guided Graph Attention Convolution Networks (GAT)??


% In this section, we take the dependency information from different layers of the dependency parser, and apply proposed parser-guided Graph Convolution Networks(GCN) to integrate the syntactic information into the basic ORL model.

% \begin{CJK}{UTF8}{gbsn}
% 理清思路再去改！！！！！！！！！！！！
% \end{CJK}

% ., and compares apply different encode approaches to integrate syntactic information to ORL model.
%parser-guided Graph Convolution Networks(GCN)
Despite of recent advances in dependency parsing \cite{dozat2016deep}, parsers still cannot output parse trees with high accuracy on out-of-domain or irregular data. In this work, we exploit rich syntactic information contained in the edge-weighted graphs to mitigate the effects of parsing errors. 
Specifically, we firstly employ graph convolutional networks (GCN) to encode the edge-weighted graphs, and then integrate them into different processing levels of ORL with implicit parser hidden states. Finally, we employ novel MTL frameworks to alleviate the error propagation problem further.
% Furthermore, considering that the pipeline architecture often lead to error propagation problem, we employ novel multi-task learning framework to exploit syntactic information from the dependency parser. 
%train a parser and a ORL model at the same time. %, simultaneously.
% In this section, we introduce the syntactic knowledge extracting from different processing levels of the parser model. Then, we apply the proposed graph dependency network to encode the edge-weighted graph. Finally, We  simultaneously integrate the structural information of the edge-weighted graph  and the obscure information of neural networks to ORL basic model under the MTL framework. 

%\subsection{Syntactic information} %In terms of utilize  syntactic information, previous works tend to only pay attention on (focus  on ) the 1-best dependency tree or the obscure SAWR. 
%In this work, we think syntactic information is contained in the above parse process, to varying degrees. 

% Different layers of the parser contain different degrees of encoded syntactic information.
%Naturally, syntactic information is contained in the parse process, to varying degrees.

% The BiLSTMs encoder extracts the neural features from the input sentence, which can be viewed as implicit and incomprehensible syntactic information, called as syntax-aware word representation (SAWR). the output of the decoder, the parsed tree, indicates the dependency relation between words of sentence and conveys explicit structural information, but the parsed tree is the highest-scoring one(hard), and seriously suffers from the error propagation of parser performance. % 这句话貌似不对，错误传播那句
% Intermediate outputs, probability matrix(the edge-weighted graph or dependency forest) should contain more and better(explicit and soft? stable (vs. SAWR)) syntactic representations. Compared with SAWR,  the edge-weighted graph gives the probability of all word pairs that hold a dependency, is more comprehensible. 
% Compared with 1-best tree, the edge-weighted graph can obtains all latent parse trees, which conveys richer structural information and alleviates the affect of error propagation, to the certain extend.


% The biaffine dependency parser model is composed of the BiLSTM encoder, and the biaffine scorer, and MST decoder, which convey rich syntax-specific knowledge with different types \cite{dozat2016deep}.
% The BiLSTM encoder extracts the neural features from the input sentence, which is regarded as a kind of implicit neural syntactic information, called as syntax-aware word representation (SAWR, called synrep in out exp). The parsed tree, the output of the decoder,  indicates the dependency relation between words of sentence and conveys explicit structural syntactic knowledge. The intermediate layer, the biaffine scorer, computes  the probability of all word pairs that hold a dependency, which represent syntactic knowledge by a edge-weighted graph.

%\subsectIon{Integration approach} In this work, we rely on  multi-layer Graph Convolution Networks (GCN) to encode the dependency information of the fully connected graph with edge weight.

% \iffalse
% \begin{figure}
%   \centering
%   \includegraphics[width=.45\textwidth]{softgcn2.pdf} 
%   \caption{Soft GCN.}  
%   \label{fig:ucca}
% \end{figure}
% \fi

\subsection{Dependency Graph Convolutional Networks (\textsc{DepGCN})}

In this subsection, we propose dependency graph convolutional networks (\textsc{DepGCN}) to better encode the syntactic information from the edge-weighted graphs. %dependency parser. 
On the one hand, compared with explicit 1-best parse trees, edge-weighted graphs convey richer structural information by providing all latent syntactic structures, and avoid error propagation as well. 
On the other hand, compared with the implicit hidden states of the parser encoder \cite{zhang2019syntaxMT,xia2019mtl}, an edge-weighted graph, denoted as an attention matrix, explicitly captures the modification strength of word pairs. 

%这两句话的逻辑可以再琢磨一下。前面有没有把hidden state方法作为一个典型的方法来介绍？
%这一句话感觉没有说透GCN的优势。

%Considering the versatility and availability of graph convolutional networks (GCN) on graph-structural data \cite{kipf2016semi}, we apply a multi-layer (GCN) to encode the syntactic information from the edge-weighted graph, and we name it as the DepGCN approach.

% Compared with the parsed tree, the edge-weighted graph can obtains all latent parsed trees including the parsed tree, which conveys richer structural information. Besides the edge-weighted graph alleviates the affect of parser performance since the gold tree is always included in it.
% Compared with the SAWR, the edge-weight graph can be viewed as attention matrix, which  indicates the dependency relation of all word pairs and is  more comprehensible.
% We employ multi-layer GCN to  to encode the structural knowledge of the edge-weighted graph produced by the parser, called Graph dependency network (DepGCN).
%Since syntactic GCNs produce representations at word level, it is straightforward to use them as encoders within the attention-based encoderdecoder framework.  In each GCN layer, information ﬂows along edges of the graph; in other words, each node receives messages from all its immediate neighbors. 
%\subsubsection{GCN}
%the weighted fully directed graph

The original GCN is designed for directly modeling graph-structured data \cite{kipf2016semi}. Although each node only receives information from its immediate neighbors through edges in one GCN layer, multi-layer GCN can propagate information more globally if there exist connected paths. 
% when the layer number is enough.
%since each node has  includes information from their respective neighbors layer by layer. 
Formally, the output of node $i$ at the $l$-th layer of GCN is computed by the following equation:
\begin{equation}\label{eq:gcn}
\begin{split}
  \mathbf{h}_i^{(l)} &= F\left(  \sum_{j=1}^n \mathbf{A}_{ij} \mathbf{W}^{(l)}\mathbf{h}_j^{(l-1)}+\mathbf{b}^{(l)} \right)
\end{split}
\end{equation} 
%probability matrix
where $\mathbf{A}$ is the adjacency matrix of a graph with $n$ nodes, $\mathbf{W}^{(l)}$ and $\mathbf{b}^{(l)}$ are the model parameters, $F$ is an activation function. $\mathbf{h}_i^{0}$ is the initial input vector.

\iffalse
\begin{figure}
  \centering
  \includegraphics[width=.45\textwidth]{multilayer-softgcn3.pdf} 
  \caption{multi-layer GCN.}  
  \label{fig1}
\end{figure}
\fi

\iffalse
\begin{figure}
  \centering
  \includegraphics[width=.45\textwidth]{orl-structure.png} 
  \caption{multi-layer GCN.}  
  \label{fig-new}
\end{figure}
\fi
%We apply multi-layer Graph Convolution Networks (GCN) to encode the dependency information of the edge-weighted graph produced by the parser, called Graph dependency network.
% As shown in Figure xx, we stack our DepGCN on the top of the BiLSTM encoder of ORL and the biaffine scorer of dependency parser, and fed the outputs of the top BiLSTM encoder and  the edge-weighted graph  as the adjacency matrix $A$ and initial inputs $\mathbf{h}_i^{0}$ of GCN, respectively. Following \cite{marcheggiani2017encoding}, we also add a self-loop for each node in the graph.
As shown by Figure \ref{fig:orl-parser}-(e), we apply \textsc{DepGCN} to connect the parser model and the ORL model. We first obtain the edge-weighted graph from the decoder of a well-trained biaffine parser as a data preprocessing step, and then feed the graph into our \textsc{DepGCN} in the form of an adjacency matrix $A$ \footnote{Moreover, following \citet{ marcheggiani2017encoding}, we also add a self-loop for each node in the graph, which means all diagonal elements of $A$ are set to 1.}. 
Then we feed the outputs of the ORL BiLSTM-based encoder as the initial inputs $\mathbf{h}^{0}$ to the \textsc{DepGCN}. Finally, we feed the output of the \textsc{DepGCN} to the CRF-based decoder, and update the ORL results under the guidance of the syntactic information. 
%we stack our DepGCN on the ORL BiLSTM-based encoder and
%which can be naturally employed to encode the edge-weight graph produced by the parser. .%instead of directly using word embeddings as its inputs. (as inputs and try)

 % Following Marcheggiani and Titov (2017), we also add a self-loop for each node in the tree. 


%For example, the input vector for node $k$ in GCN is $x^{(0)}_i &= r^{bilstm}_i}$, where $r^{bilstm}_i$ is the output of the top of BiLSTM layer of ORL.（xx 讨论了GCN层数和位置的影响。）

%(It is worth noting that  they also use GCN to encode a fully connected edge-weighted graph which produced from the best dependency tree  by using self-attention mechanism, while the fully connected edge-weighted graph we used is produced by the dependency parser),
Moreover, we introduce dense connections to the multi-layer \textsc{DepGCN} for extracting more structural information \cite{huang2017densely,guo2019attention}. Instead of only adding connections between adjacent layers, we use dense connections from each layer to all the subsequent layers. Formally, the input of node $i$ at the $l$-th layer is:
\begin{equation}\label{eq:dense}
\begin{split}
    \mathbf{x}_i^{(l)} = \mathbf{h}_i^{(0)} \oplus \mathbf{h}_i^{(1)} \oplus \cdots \oplus \mathbf{h}_i^{(l-1)}
\end{split}
\end{equation}
where $\mathbf{h}_i^{(l)}$ is the output of node $i$ at the $l$-th layer. We also make residual connections over \textsc{DepGCN} to mitigate the vanishing gradient problem, which means that the output dimension of each \textsc{DepGCN} layer is decided by the layer number and the input dimension of the bottom \textsc{DepGCN}.


% Although GCN has a wide range of versatility, 

% \textbf{VS. TreeLSTM.} As a generalization of the standard LSTM architecture, TreeLSTM is proposed  for encoding tree-structured data, especially for dependency tree. Compared with the versatility of GCN, the application scenario of TreeLSTM has limitations, since it can not process graph-structured data since the loop connection or non-connection.  Many previous  works focus on employing TreeLSTM to obtain the structural information of syntax tree. Give a dependency tree, TreeLSTM  propagates information
% from the root node to the node of leaves, or sends information in the opposite direction, while GCN only encode information from its neighbors. From this perspective, it seems show  that the superiority of TreeLSTM  for representing the dependency tree over GCN. Considering the performance of paring, GCN  only using information about the substructure of the tree is better than TreeLSTM that globally rely on the whole tree. In this work, We would compare the effects of GCN and TreeLSTM encoding syntax trees through experiments. 
% %Furthermore, \citet{xia2019syntax}  Simultaneously use all  outputs of the multi-layer BiLSTM layer  by the normalized weighted-sum operation. 
% % v

% \textbf{VS. hidden states.} In order to avoid the error propagation problem induced from syntax parser, Predecessors propose to extract syntactic information from the encoder of a pre-trained parser, and use the output vectors of the encoder as the syntax representations at word level\cite{xia2019syntax, zhang2019syntax}, Called SynRep (SAWR) approach. 
% But this method simply view high-level neural features as syntax representations, which is too obscure to make a reasonable explanation. On the one hand,  parser-guide GCN uses the information of all possible tree, not only the highest-scoring tree, which compensates for the parsing inaccuracy in a certain extent. On the other hand, the edge-weight graph means how possible all word pairs hold valid dependency relations, which the probability is explainable.

% \textbf{VS. other methods.} 
% DEPHead
% TPF
% GCN-hard

%\citet{marasovic2017srl4orl} propose different MTL technology to improve the performance of ORL with  simultaneously training with SRL. 
% since this  kind  of sharing strategy somewhat weakens the representation framework maintains distinct model parameters  for  each  task, \citet{xia2019syntax} propose to integrate the lstm hidden states of the parser into the input layer of the basic SRL model, which achieve better performance.
%and naturally extend to GCN-soft.
% \textbf{VS. integration of parse tree}

% DEPHEAD
% TPF
% TreeLSTM
% DepGCN-Hard
% \textbf{VS.  integration of SAWR}
% as inputs, please read Multi-task learning section for integration details. 

% The edge-weighted graph and parser hidd   en states deliver different syntactic information, respectively.
% The neural syntactic feature contained in parser hidden states
\subsection{Combining Explicit and Implicit Syntax (\textsc{DepGCN+DepHDN})}

Different from explicit 1-best parse trees or edge-weighted graphs, hidden states of the BiLSTM encoder of a dependency parser provide useful syntactic knowledge and are less sensitive to parsing errors. Using such implicit syntactic representations has been demonstrated to be highly effective for downstream tasks \cite{zhang2019syntaxMT,xia2019mtl}. In this section, we describe how to integrate implicit syntactic information from parser hidden states and explicit syntactic information from the edge-weighted graph into the ORL model for better performance.

%We introduce the DepHidden approach in details.
% Firstly, we integrate the parser hidden states into the input layer of the ORL as extra word representations. 
% In particular, we use the encoder of a well-trained parser as a neural syntactic information extractor, which generates syntactic representations of the ORL data at word level.
We first briefly describe the use of the dependency parser's hidden states, named as \textsc{DepHDN}. As shown by Figure \ref{fig:orl-parser}-(d), we extract the outputs from the parser encoder and feed them into the BiLSTM-based encoder of ORL as extra inputs. The hidden states of each parser BiLSTM layer are obtained as the syntactic representations, i.e., $\mathbf{h}_1^{(l)},\cdots, \mathbf{h}_n^{(l)}$, where $\mathbf{h}_n^{(l)}$ is output of the $l$-th layer of the parser BiLSTM encoder at $w_n$. Then, we use the weighted-sum operation to get a single vector $\mathbf{h}^{syn}_i$ as the final syntactic representation of word $w_i$. 
\begin{equation}\label{eq:weighted-sum}
\begin{split}
    \mathbf{h}^{syn}_i = \mathbf{W}\lambda \sum_{j=1}^{L} \alpha_j \mathbf{h}_i^{j}
\end{split}
\end{equation}
% , $h_i^j$ is the output of $j$-th layer of parser encoder at word $w_i$
%that is detailedly introduced in the above subsection
where $L$ is the layer number of parser BiLSTM-based encoder; $\mathbf{W}$, $\alpha_j$ and $\lambda$ are model parameters; $\alpha_j$ is softmax-normalized weights for $h^j$; $\lambda$ is used to scale the syntactic representations. The syntactic representations $\mathbf{h}^{syn}_i$ are concatenated with the original ORL input vectors, so that richer word representations are obtained.
% which we refer as implicit syntactic information integration. %, which implicit parser hidden states have been integrated into ORL model so far. 

Furthermore, in order to simultaneously benefit from the implicit and explicit syntactic information, as shown in Figure \ref{fig:orl-parser}-(f), we simply extract the edge-weighted graph from the parser decoder and apply the \textsc{DepGCN} approach over the ORL encoder to obtain syntax-enhanced representations.

\subsection{Pipeline vs. Multi-Task Learning }
\iffalse
\begin{figure}[tb]
    \centering
    \includegraphics[width=.45\textwidth]{mtl.png}
    \caption{Multi-task learning.}
    \label{fig:fig-mtl}
\end{figure}
\fi


The three approaches, depicted in Figure \ref{fig:orl-parser}-(d-f) respectively, can work either in the pipeline way or in the MTL way. 
Specifically, the pipeline way first trains the dependency parser and then fixes the parser components during training the ORL model. 
In contrast, the MTL way trains both the parser and the ORL model at the same time.  
In this subsection, we explore the MTL way to  alleviate the error propagation problem further besides the \textsc{DepGCN} approach.  


%In this section, we employ multi-task learning framework to further alleviate the error propagation problem, while the edge-weighted graph and parser hidden states has been less sensitive to parsing errors to a certain extent.
As a baseline, Figure \ref{fig:orl-parser}-(c) shows the most common MTL method, which shares a common encoder and uses multiple task-specific output layers, known  as the hard-parameter-sharing MTL \cite{ruder2017overview,marasovic2017srl4orl}. However, this approach is not suitable for our scenario where the auxiliary parsing task has much more labeled data than the main ORL task, since the shared encoder is very likely to bias toward to parsing performance \cite{xia2019mtl}. 
% (ORL) ails to maintain (enough) distinct parameters for each task \cite{xia2019mtl}, especially for . 
%weakens the representation of the framework that maintains distinct model parameters for each task 

%which keep own model parameter  without sharing any layers' parameters
Inspired by \citet{xia2019mtl}, we adopt the architectures of Figure \ref{fig:orl-parser}-(d-f) to keep task model parameters separately, and train ORL and the parser simultaneously. We update model parameters according to the combined loss of the ORL and the dependency parser during training: 
\begin{equation}\label{eq:loss}
\begin{split}
    \zeta &= \zeta_{ORL} + \alpha \zeta_{Dep}
\end{split}
\end{equation}
where $\zeta_{ORL}$ and $\zeta_{Dep}$ is the loss of the ORL model and the parser respectively, and $\alpha$ is a corpus weighting factor to control the loss contribution of the dependency data in each batch as discussed in Section \ref{sec:setting}. 

Compared with the previous pipeline training process, the parameters of the parser are not pretrained and fixed, but updated by training objectives of both ORL and the parser. This results in a ORL-preferred dependency parsing model. 

% We  view the dependency parser as our syntactic information extractor, which generates parser hidden states and edge-weighted graph for DepHidden and DepGCN approaches as shown by sub-figure b, c in Figure \ref{fig:fig-mtl}, respectively. The difference is that ORL data can obtain different parser hidden states and edge-weighted graph at each training step, whereas the above pipeline architecture only can provide same syntactic information in the whole training process. Besides, we also can simultaneously integrate 


% For ORL Module, we firstly feed ORL data into the parser module, and then obtain the hidden states  from parser encoder and edge-weighted graph from the parser biaffine scorer for DepGCN and DepHidden approaches. The difference is that ORL data can obtain different parser hidden states and edge-weighted graph at each training step, whereas the above pipeline architecture only can provide same syntactic information in the whole training process. 

% For the parser module, in addition to be updated by loss from parser, the model parameters  would also be updated by the loss from the ORL, which learn a ORL-preferred dependency parsing model.
 %i.e., hard/soft parameter sharing.
 
% 多任务框架下可以实现以上三种方法。 这里我们在多任务框架下同时使用1，2信息，考虑到：
% 1）森林中有gold tree
% 2) 
% 3) 
% In order to mitigate errors propagation, we investigate to train the biaffine parser and the ORL model at the same time.

% In order to alleviate the error propagation problem, we integrate different syntactic information into the ORL model under the MTL framework, i.e., the hidden states of the parser encoder and the edge-weighted graph.

% Multi-task learning has led to successes in many applications of NLP by incorporating labeled data of multiple related tasks. 

% There are several widely-used approaches to perform MTL in deep neural networks. The most common method is to share several hidden layers of the encoder between tasks and keep the task-specific output layers, which is called hard-parameter-sharing MTL \cite{ruder2017overview,marasovic2017srl4orl}. However, this method weakens the representation framework and maintains distinct model parameters for each task, especially for the tasks with different sizes of labeled data \cite{xia2019mtl}.

% The hard parameters sharing MTL, the  most commonly used strategy, shares several hidden layers between tasks and keeps the task-specific output layers \cite{ruder2017overview}, which weakens the representation framework maintains distinct model parameters  for  each  task, especially for the tasks that has big gap in the size of labeled data \cite{xia2019mtl}.

%Consider the above problem, \citet{xia2019mtl} integrate the LSTM hidden states of the parser into  the basic SRL model, which achieve better performance than hard parameters sharing MTL.
% As shown in figure xxx,   First, we usewe extract all the 3 BiLSTM hidden outputs of the dependency parser as the syntactic representations. Second,we employ the normalized weights to sum the extracted representations as the ﬁnal syntactic representation for word $w_i$, denoted as $rep_i^{syn}$. Formally, xxxxxxxxxxx. the extracted syntactic representations are fed into the input layer of the SRL module, and concatenated with the original SRL module input.

% Multi-task learning (MTL) has led to successes in many applications of NLP by incorporating labeled data of multiple related tasks. There is several commonly used ways to perform MTL in  deep neural networks, and the hard parameter sharing is the most commonly used, which shares several hidden layers between tasks and keeps the task-specific output layers. 
% However, the hard sharing strategy somewhat weakens the representation framework maintains distinct model parameters  for  each  task, especially for the tasks that has big gap in the size of labeled data.

% Inspired by \citet{xia2019mtl}, we apply the novel MTL framework to  train ORL and paser model at the same time, and naturally combine the structural information of the edge-weighted graph  and the obscure information of the hidden states of the parser to ORL basic model.  

% we can view the dependency parser as our syntactic information extractor, which generates syntactic representations of the ORL data. % of parser BiLSTM encoder and edge-weighted graph for ORL data.
% As shown in Figure \ref{fig:orl-parser}, we feed the ORL data into the BiLSTM encoder of the dependency parser, and obtain hidden outputs of each hidden layer as the syntactic representations, i.e., $h_1^{(l)},\cdots, h_n^{(l)}$, where $h_n^{(l)}$ is output of the $l$-th layer of the parser BiLSTM encoder at $w_n$. Then, in order to make full use of all hidden outputs of each word, we use weighted sum operation to get a single vector as the syntactic representation as the following equation:
% \begin{equation}\label{eq:weighted-sum}
% \begin{split}
%     h^{syn}_i = W\lambda \sum_{j=1}^{L} \alpha_j h_i^{j}
% \end{split}
% \end{equation}
% where $h^{syn}_i$ is the ﬁnal syntactic representation for word $w_i$, $W$ is a model parameter, $h_i^j$ is the output of $j$-th layer of parser encoder at word $w_i$, and $L$ is the layer number of parser encoder, and $\alpha_j$ is softmax-normalized weights for $h^j$, and the scalar parameter $\lambda$ can scale the syntactic representations; both $\alpha$ and $\lambda$ are updated during training. Finally, the syntactic representations $h^{syn}_i$ are concatenated with the original input the ORL model as extra word embeddings. 

% Apart from integrating the hidden states of parser encoder, we apply the DepGCN approach over the ORL encoder to obtain syntax-enhanced representations, i.e., the edge-weighted graph, so that the MTL model simultaneously combines the implicit and structural syntactic information (Figure \ref{fig:orl-parser}).
% We design this combination for several considerations:
% 1) More types of syntactic information are used, both obscure and interpretable  structural information.
% 2) LSTM encoder preserve sequence information over time that weakens the structural information of the syntax. Soft-GCN stacked over the LSTM can obtain a better syntax-enhance (syntactically enhanced) word representation.
% \begin{CJK}{UTF8}{gbsn}
% 1) 使用了更多的类型的句法信息，既有隐晦的信息，也有可解释的结构信息
% 2) lstm编码是一种时间序列的编码，打乱（削弱）了句法的结构信息，在LSTM基础上加上Soft-GCN，获得更好的句法增强的词向量表示。
% \end{CJK}

% Following by \citet{xia2019mtl},  we update model parameters according to the combined loss of the ORL and the dependency parser when the training,:
% \begin{equation}\label{eq:loss}
% \begin{split}
%     \zeta &= \zeta_{ORL} + \alpha \zeta_{Dep}
% \end{split}
% \end{equation}
% where $\zeta_{ORL}$ is the loss of ORL task, $\zeta_{Dep}$ is the loss of biaffine parser, and $\alpha$ is a corpus weighting factor to control the loss contribution of the dependency data in each batch as discussed in the following experiment setting (Section \ref{setting}).

% the proposed framework keeps the own model parameters for each task,thereby maximizing task-speciﬁc information for the main task
% the dependency parser module can be updated by the gradients returned from the extracted syntactic representations,which can encourage it to produce semantic preferred representations.

% \subsection{GCN}

% structural information. GCN encoder, stack on lstm of orl
% \textbf{Hard GCN.}
% a viterbi-decoded tree

% \textbf{soft GCN.}
% prob soft,....

% \subsection{combination}
% simple combination

% \subsection{MTL}
% task-specific parser.

%• The TPF (tree position feature) approach considers the
% relative positions of wi and wp to their common ancestor
% word in the parse tree, and use the embedding of such
% position features as the representation of wi
% .