\section{Model}

\subsection{The Label-text Feature}
\label{feature}
We discover that label descriptions can provide precise cues to classify a document which contains \textit{multiple} semantic aspects. Figure \ref{fig:label} provide an example on the necessity of integrating label description. It should be noted that when patent examiners classify documents to the CPC system, they are also advised to use cue words and to search among label descriptions \footnote{see IPC guidelines at \url{https://www.wipo.int/export/sites/www/classifications/ipc/en/general/guidelines_where_to_classify.pdf}}.

\begin{figure}
    \centering
    % \includegraphics[width=0.4\textwidth]{ipc.png}
    % \caption{Illustration of the CPC hierarchy.}
    % \label{fig:ipc}
    % \vspace{0.2cm}
    \includegraphics[width=0.4\textwidth]{Label_5.png}
    \caption{Example of the label description (left boxes) and classification cues (shadowed texts). The true label for the input text (right box) is B01D. Although the given text discussed neural networks, which is semantically closer to G06N, the classification cue of "ultrafiltration" decides for  B01D eventually. }
    \label{fig:label}
    \vspace{-0.3cm}
\end{figure}

We integrate the label information through the label-text feature that captures the string-level relatedness between label descriptions and texts. Here we use BM25 score:.

\begin{equation} \label{eq:bm25} 
\resizebox{.87\hsize}{!}{\text{BM25}(D_k,x) = \sum_{i=1}^{n} \text{idf}(x_i) \frac{\text{tf}(x_i, D_k)(k_1 + 1)}{\text{tf}(x_i, D_k) + k_1(1 - b + b \frac{|D_k|}{\text{avgdl}})}}
\end{equation}

where $x_i$ is the $i$th item in text x; $\boldsymbol{D}_k$ is the label description for class $k$; \textit{idf} and \textit{tf} are inverse document frequency and text frequency for $x_i$ in $D_k$ respectively; \textit{avgdl} is the average description length; $|D_k|$ is the document length; $k_1$ and $b$ are hyper parameters. 


\subsection{Wide and Deep Structure} 
We adopt the Wide and Deep (WnD) neural network \cite{Cheng2016} for text classification . Given a training set ${(\boldsymbol{X}_n, \boldsymbol{D}, \boldsymbol{y}_n)}_{n=1}^N$, where $\boldsymbol{X}_n$ is the input document texts, $\boldsymbol{D}$ is the input label descriptions and $\boldsymbol{y}_n$ is the true labels. The model outputs the probabilities for each of the $K$ classes $\boldsymbol{\hat{y}} \in R^K$. The training target is to minimize binary cross entropy loss: $L = \frac{1}{N}\frac{1}{K}\Sigma_{n=1}^N\Sigma_{k=1}^K\text{CE}(\boldsymbol{y}_n^{(k)}, \boldsymbol{\hat{y}}_n^{(k)})$,
% \begin{equation}
%   L = \frac{1}{N}\frac{1}{K}\Sigma_{n=1}^N\Sigma_{k=1}^K\text{CE}(\boldsymbol{y}_n^{(k)}, \boldsymbol{\hat{y}}_n^{(k)}) 
% \end{equation} 
where $\boldsymbol{y}_n^{(k)}$ is the $k$th element in vector $\boldsymbol{y}_n$.

An overview of the WnD classifier is shown in figure \ref{fig:model_overview}. The model has two parts: the wide part and the deep part. The wide part takes in the label-text feature to capture string-level relatedness between the label descriptions and the text; the deep part maps the input text to word embeddings and go through a non-linear transformation to capture semantic-level relatedness between the text and the label. 
\begin{figure}
    \centering
    \includegraphics[width=0.47\textwidth]{model_overview_9.png}
    \caption{WnD classifier structure. The Deep part (left) captures semantic information; the Wide part (right) captures label-text relatedness. The label-text feature is a $K$-dimensional vector, where $K$ equals to the label set size.}
    \label{fig:model_overview}
    \vspace{-0.3cm}
\end{figure}

\textbf{The Wide part}: The Wide part is a regression model with the form $\boldsymbol{\hat{y}}_{wide} = \sigma(\boldsymbol{W}_{wide}^T\boldsymbol{z}_{wide} + \boldsymbol{b})$, where $\boldsymbol{z}_{wide}$ is the label-text interaction features as described in section \ref{feature}: $\boldsymbol{z}_{wide} = \text{BM25}(\boldsymbol{D}, \boldsymbol{X_n})$.

\textbf{The Deep part}: The Deep part is a non-linear transformation of the input text that aims to capture the semantic of the text. It can be a classic neural network for text encoding, such as RNN, CNN, or simple fully connected network. In this paper, we use textCNN \cite{Kim2014} for the transformation because it is a simple baseline that works reasonably well. The Deep part transform the texts to a fixed-length representation $\boldsymbol{z}_{deep}$. The representation $\boldsymbol{z}_{deep}$ is then mapped to $K$ classes using sigmoid activation $\boldsymbol{\hat{y}}_{deep} = \sigma(\boldsymbol{W}_{deep}^T\boldsymbol{z}_{deep} + \boldsymbol{b})$.  

The Wide and Deep parts are concatenated at the top and are jointly trained through $\boldsymbol{\hat{y}} = \sigma(\boldsymbol{W}_{deep}^T\boldsymbol{z}_{deep} + \boldsymbol{W}_{wide}^T\boldsymbol{z}_{wide} + \boldsymbol{b})$ in order to let the semantic and the string level relatedness complements each other when making the decision. In this way, the model simulates the behavior of patent examiners classifying a document: when they are uncertain which labels to assign (i.e. when semantic knowledge cannot provide a certain answer), examiners will resort to searching for cue words in the label descriptions for a clue.



% \subsection{Label-Text Relatedness Feature}\label{feature}
% As mentioned above, the Wide part should capture the label-text relatedness. We choose BM25 score between the text and label descriptions for this purpose.The wide feature is thus 

% BM25 is an effective relevance measure for document retrieval. By incorporating the BM25 shallow feature, the model simulates the behavior of patent examiners classifying a document: when they are uncertain which labels to assign (i.e. when semantic knowledge cannot provide a certain answer), examiners will resort to searching for cue words in the label descriptions for a clue.








