\section{Introduction}\label{Intro}
%%%%%%%%%% << Reading strategies of successful readers through the three phase approach >>

Automatic text summarization has wide popularity in NLP applications such as producing digests, headlines and reports. Among the supervised methods, two main types are usually explored, namely abstractive and extractive summarizations~\cite{nenkova2011automatic}.
%Abstractive approaches generate new words and phrases based on original contexts with various text rewriting operations, which is in fact a sequence mapping task. Sequence-to-sequence learning~\cite{kalchbrenner2013recurrent,sutskever2014sequence,cho2014learning} is commonly used in basic abstractive models which contain an encoder and a decoder. Attention mechanism has been broadly used in seq2seq models in which the decoder generates sequence based on the attended information from the encoder~\cite{bahdanau2014neural,luong2015effective}. With the advantages of being concise by performing generation, many attention-based seq2seq models~\cite{rush2015neural,chopra2016abstractive,nallapati2016abstractive} have been proposed. \cite{zhou2017selective} considers alignment relationship and encoder noise between the source context and the target summary. \cite{lin2018global} has proposed attention-based seq2seq models to improve repetition and semantic irrelevance problems.
%Abstractive approaches generate new words and phrases based on original contexts with various text rewriting operations, which is in fact a sequence mapping task,
%while extractive systems produce summary by extracting and copying text snippets from scratch and concatenating them into a shorter version document. 
Compared with abstractive approaches, extractive methods are more practical and applicable as they are faster, simpler and more reliable on grammar as well as semantic information~\cite{yao2018deep}.
%Furthermore, extractive systems can be applied to various fields instead of relying too much on training contexts.

Recent studies~\cite{cheng2016neural,nallapati2017summarunner,yasunaga2017graph,feng2018attentive} consider extractive summarization as a sequence labeling task, where each sentence is individually processed and determined whether it should be extracted or not.
% Firstly, sequentially extractive labels are generated by heuristic methods, in which case the training is limited by the quality of heuristic methods.
Various neural networks are used to label each sentence and trained using cross-entropy loss to maximize the likelihood of the ground-truth labeled sequences, which may derive the mismatch between the cross-entropy objective function and the evaluation criterion.
On the other hand, some reinforcement learning based methods~\cite{wu2018learning,narayan2018ranking,yao2018deep} directly optimize the evaluation metric by combining cross-entropy loss with rewards and train model with policy gradient reinforcement learning.
Note that the rewards usually reflect the quality of extracted summary and measured by standard evaluation protocol.
% \redfont{by combining cross-entropy loss with rewards from summary evaluation measures and train model parameters with policy gradient reinforcement learning}.
However, they still sequentially process text and tend to extract earlier sentences over later ones due to the sequential nature of selection~\cite{dong2018banditsum}.

% considers extractive summarization as a contextual bandit to extract summary-worthy sentences, which requires fewer update steps but lacks flexibility in extracting various numbers of sentences. 

% \redfont{\reminder{a suggestion: the flow of this paragraph. Most of previous methods adapt sentence labeling for extractive summarization. they process sentences individually and determine whether it should be extracted or not. the dismatch of cross entropy and rouge, etc. Then, some RL based methods optimized ROUGE, however, hard to balance explore and exploit and still each sentence individually.}}
%Thirdly, previous methods form sentence representation with RNN-based structure and shallowly capture the relation among all the sentences. More importantly, human beings’ reading cognitive process has rarely been explored to help extractive summarization.

\begin{figure*}[!htbp]
    \centering
    \includegraphics[width=16cm,height=5cm]{figure/intro.eps}
    \caption{An example of how human beings extract summary. The article is from CNN/DailyMail dataset.}
\label{fig:article_case}
\end{figure*}


Although great efforts have been devoted to this field, most of the existing approaches neglect how human being reads and forms summaries.
Human beings are very good at refining the main idea of a given text based on their reading cognitive process.
Note that the reading habits of native speakers are varied and hard to modeled, so we adopt the three reading phases of second-language readers where there are potential behavior patterns.
Such reading process of second-language readers
generally includes \emph{pre-reading}, \emph{reading} and \emph{post-reading}~\cite{avery1997scaffolding,saricoban2002reading,toprak2009three,pressley2012verbal}.
In the \emph{pre-reading} stage, they roughly preview the whole text to form an initial cognition and extract general but coarse-grained information at the meantime.
Based on such prior knowledge, the subsequent \emph{reading} stage is a conscious process that focuses on target-specific purposes to search fine-grained details through repeated skimming and scanning.
For \emph{post-reading}, re-reading is performed to check whether there are any missed details.
The three-stage reading process makes it effective in capturing essential sentences of text as the extracted summarization.



%above issues indicate the limitation of existing structures to flexibly predict the distribution of importance on various texts and form summary.
%However, human being is good at capturing most informative sentences for each document based on his reading cognitive process which has rarely been exploited for extractive text summarization.
%In practical situation, there are three stages for effective reading, including pre-reading, reading and post-reading~\cite{avery1997scaffolding,saricoban2002reading,toprak2009three,pressley2012verbal}.
%In the pre-reading stage, human beings roughly preview the whole text to form an initial reading cognition and extract general and coarse-grained information in the meantime.
%Based on prior knowledge, reading is a conscious process through repeated skimming and scanning, and it focuses on the target-specific purpose to search for fine-grained details.
%For post-reading, human beings re-read to understand the context deeply and make corrections for earlier cognition.
%Three-stage reading process is effective in capturing the main idea and extract target-specific information on purpose.

Inspired by such human's reading cognitive process, in this paper, we re-examine the problem of extractive summarization and propose a new approach \textbf{HER}~(\textbf{H}uman-b\textbf{E}ing-\textbf{R}eading inspired extractive summarization). 
% It consists of two subsequent stages called \emph{rough reading} and \emph{careful reading}.
We simplify the three-stage reading process to two subsequent stages called \emph{rough reading} and \emph{careful reading}. 
In \emph{rough reading}, coarse-grained information of the original context is identified to form a general cognition.
A detailed case is shown in Figure~\ref{fig:article_case}.
In Figure~\ref{fig:article_case}~(a) and (b), after browsing on the whole article, the main idea is outlined and the text is roughly divided into three parts based on the gist of paragraphs at the meantime. Each part describes related but not the same contents.
To implement the \emph{rough reading} process, we use a hierarchical neural network to encode sentence vectors and derive a document representation as global feature for the main idea.
Meanwhile, a convolutional neural network~(CNN) is utilized to encode local features from different paragraphs.

During \emph{careful reading}, the model searches for specific but important details through re-readings to cover the content and extract essential fine-grained information as the final summary.
For instance, as shown in Figure~\ref{fig:article_case}~(c), after rough reading, two sentences close to the main idea ``Boston Bruins won Stanley Cup'' may be selected firstly.
Then an earlier and more detailed sentence about ``fans rioting'' is appended to the summary by performing re-reading. It is a combination of people's \emph{reading} and \emph{post-reading} process.
To accomplish this, we train a neural network to score each sentence.
% called sentence affinity 
% which can be considered as an estimation of sentence correlation to cover the context.
A multi-armed bandit policy with an adapted termination mechanism is then used to form the final summary.

In our HER model, the whole process is formulated as a contextual bandit problem. We train a reinforcement learning agent to solve it using the policy gradient method~\cite{sutton2000policy}.
At each step, the agent takes an action which is a to-be-selected sentence set, and then receives a reward based on the correlation between extractive summary and gold-standard reference summary.

Our main contributions are as follows:
\begin{itemize}
	\item We propose an extractive summarization method that simulates human being's reading cognitive process. We formulate it as a contextual bandit problem in which two stages including rough reading and careful reading are devised.
%	Inspired by human being's reading cognitive process, we present a two-stage method containing rough reading and careful reading for extractive summarization, which is generally formulated as a contextual bandit.
	%\item In rough reading, a hierarchical neural network is used to generate an overall representation and a CNN-based network is adopted to capture local information.
	\item We use a hierarchical neural network for rough reading which consists of a neural net to encode the whole document and another one to capture features in paragraphs. Then we use a contextual-bandit agent to flexibly select sentences during careful reading, with an adapted termination mechanism to select various but proper numbers of sentences.
	%\item We propose a novel approach called SHE formulated as a contextual bandit and trained with reinforcement learning.form summaries of less redundancy and more high quality and significantly
	\item We conducted experiments on the CNN and DailyMail datasets and showed that our proposed model can outperform state-of-the-art methods and provide high-quality summaries.	
\end{itemize}
 