\subsection{Benchmarking Approach}

We establish a baseline for abstractive summarization by using the neural summarization architecture introduced by \cite{chen-bansal-2018-fast}\footnote{\href{https://github.com/ChenRocks/fast_abs_rl}{github.com/ChenRocks/fast\_abs\_rl}}. The generated data has noise due to imperfections in the alignment method and due to potentially broken coreference, so we use the model in a semi-supervised fashion.


We choose this architecture as a baseline for several reasons: (1)~The paradigm for narrative summarization from noisy dialogue is close to the paradigm assumed by Chen and Bansal. Namely, first extract salient sentences, then abstractively rewrite them with an included copy mechanism to deal with OOV words. (2)~The ability to analyze the extractor behavior separately from the abstractor due to the independence of training (before connection by the reinforcement learning mechanism). (3)~The speed of training due to the shortened input-target pairs.
%  \begin{itemize}
%  \itemsep0em 
%   \item The paradigm for narrative summarization from noisy dialogue is close to the paradigm assumed by Chen and Bansal. Namely, first extract salient sentences, then abstractively rewrite them with an included copy mechanism to deal with OOV words.
%   \item The ability to analyze the extractor behavior separately from the abstractor due to the independence of training (before connection by the reinforcement learning mechanism).
%   \item The speed of training due to the shortened input-target pairs.
%  \end{itemize}

We briefly describe the model: First, the model optimizes a sentence extraction module and an abstractive rewrite module independently using maximum-likelihood objectives. Then, end-to-end training is achieved by applying policy gradient methods (due to the ``non-differentiable hard extraction" performed by the extractor). The extractor uses a temporal convolutional model to obtain hierarchical sentence representations, then selects sentences using a pointer network. The abstractor is an encoder-aligner-decoder network with a copy mechanism for OOV words. Due to the large amount of non-narrative chit-chat turns between salient turns, we train the extractor on a sequence of turns rather than individual sentences.
% In our use of the model, we make a small change to the training paradigm where we train the extractor on a sequence of turns rather than sentences. This is due to the large amount of non-narrative chit-chat turns we find in between the salient turns.

\subsection{Evaluation and Analysis} 

We use precision, recall, and F-1 scores of ROUGE-1, 2, and L, along with METEOR \cite{denkowski2014meteor} to evaluate the generated summaries (Table \ref{tab:model metrics}). We run these metrics on the test set, using both the combined extractive-abstractive model and the purely extractive model for analysis on what turns are considered salient.

\begin{table}[!t]
\centering
\small
\begin{tabular}{lcccc}
\hline
& \textbf{R1} & \textbf{R2} & \textbf{RL} & \textbf{M}\\
\hline
\multicolumn{5}{c}{\textbf{Extractive (rnn-ext + RL)}}\\ \hline
\textbf{P} & 20.83\(\pm .34\) & \textbf{7.34\(\pm 0.28\)} & 18.38\(\pm .32\) &  \\
\textbf{R} & \textbf{44.59\(\pm .66\)} & \textbf{17.42\(\pm .62\)} & \textbf{39.22\(\pm .61\)} &  \textbf{16.58} \\
\textbf{F1} & \textbf{25.20\(\pm .34\)} & \textbf{9.23\(\pm .32\)} & \textbf{22.20\(\pm .32\)} & \\
\hline
\multicolumn{5}{l}{Reported Metrics on CNN/DM}\\
\textbf{F1} & 41.47 & 18.72 & 37.76 & 22.35\\
\hline
\multicolumn{5}{c}{\textbf{Abstractive (rnn-ext + abs + RL + rerank)}}\\ \hline
\textbf{P} & \textbf{27.38\(\pm .34\)} & 5.91\(\pm .20\) & \textbf{25.18\(\pm .32\)} & \\
\textbf{R} & 22.65\(\pm .27\) & 4.75\(\pm .16\) & 20.74\(\pm .26\) & 8.33\\
\textbf{F1} & 23.35\(\pm .23\) & 4.91\(\pm .16\) & 21.41\(\pm .23\) &\\
\hline
\multicolumn{5}{l}{Reported Metrics on CNN/DM}\\
\textbf{F1} & 40.88 & 17.80 & 38.54 & 20.38\\
\hline
\end{tabular}
\caption{ROUGE (Precision, Recall, F1 \(\pm\) 0.95 conf. interval) and METEOR (M) metrics on the \crd~ test set using the purely extractive and extractive+abstractive architecture proposed by Chen and Bansal. We show the metrics on the CNN/Daily Mail dataset for the same models as reported by Chen and Bansal.}\label{tab:model metrics}
\end{table}

The purely extractive model significantly outperforms the combined model in recall and in F-1, due to the much higher recall. In the validation set, we notice the recall measures are improved by the n-grams in summary chunks that have indirect speech (``fjord says", ``he says", etc). In the validation set, the mean ratio of unique overlapping summary n-grams to total unique summary n-grams are: 1-gram= 0.679, 2-gram= 0.336, and 3-gram= 0.205. This high rate of 3-gram overlap motivates changes to the modeling architecture that are more lenient towards phrasal copy instead of just enabling word copy and depending on the learned language model and the word level copy probability.

\begin{figure}[!t]
\centering
\small
\begin{tabular}{p{6.6cm}}
\hline
\textbf{Generated Abstractive Summary}\\
\hline
he says he feels worried about trying to learn about these abilities and abilities . \\
he asks if she could try and cheer . \\
the group then heads to the tavern . \\
she asks if she can see jester , and she says she 's really disturbing . \\
\hline
\end{tabular}
\caption{Extractor+Abstractor output for the dialogue sample in Figure \ref{sample-dialogue}}\label{tab:sample_output}
\end{figure}
The grammatical person shift and significant paraphrasing of turns lower the precision of the purely extractive model, leading to a higher precision in the combined model.
% Though the precision of the abstractive model is higher, this seems to be generally because of the grammatical person shift and significant paraphrasing of the turns which lower the precision scores of any extractive approach. 
For example in Figure \ref{sample-dialogue}, ``beau asks about jester ." from the human-authored summary is entirely from turn 1, but the only overlapping word is ``jester". From Figure~ \ref{tab:sample_output}, we can see the encoder-decoder model learns the grammatical shift behavior but doesn't include the proper nouns, so the resulting summary misses important speaker information that is included in the human generated summaries. For example, Beau is the character alias for Marisha, which is latent information that was not available to the model at the time of decoding/generation. We also note the encoder-decoder module's learned language model is biased by the narrative elements present in the training dialogue chunks. This causes decoding of similar, but fundamentally different, narrative focused turns to be noisy and nonfactual.

Compared to news summarization metrics with the same model architectures, the dialogue summarization metrics are substantially lower. 
The disparity in model performance can be attributed to content selection differences between news -- where effective summary information is available early in an article (position bias) -- and dialogue -- where the positional effects are not observed. Other factors include the grammatical and stylistic differences explored earlier.
% In addition to the grammatical and stylistic differences explored earlier, we attribute this disparity in model performance to the content selection differences between news and dialogue. 
Our findings also confirm the findings of \cite{kedzie-etal-2018-content}, which compares content selection methods for summarization across various domains (CNN/DM, NYT, DUC, Reddit, AMI, and PubMed). They find a similar disparity in R-2 (recall) and METEOR scores between the news domain and the AMI meeting dialogue domain. They also include an oracle measurement as a performance ceiling; it achieves a max METEOR score of 17.8 and R-2 recall of 8.7 on the AMI corpus. Though ROUGE and METEOR are more useful for relative measurements than absolute, we find the current evaluation methods in summarization lead to skewed and less informative scores in dialogue domains. The problem is compounded in narrative summarization due to narrative specific lexical information, including speaker aliases. For example, METEOR specifically considers synonyms, paraphrases, and function words; all of which can change a lot from narrative to narrative. 