The CRD3 dataset can be applied to many tasks, but we find abstractive dialogue summarization the most compelling task to explore in this paper. Due to the extensive length of the dialogues and summaries, and the frequent context switching and noise, we are presented with challenges that are poorly addressed by the current modeling and evaluation methods:
\setlist{nolistsep}
\begin{enumerate}
% \itemsep0em 
  \item The dataset has relatively few episodes (159); as is, this is not enough samples to train, test, and validate using current neural approaches.
  \item The current, most successful summarization approaches do not explicitly attempt to capture coreference, semantics, and pragmatics in very long documents or conversations.
  \item Current automatic summarization evaluation methods have specific failures in evaluating narrative summarization.
\end{enumerate}
We do not attempt to propose a solution for either the second or third challenges, as they are beyond the scope of this paper. Instead, we address the first challenge by proposing a novel data augmentation method to dramatically scale up the number of available summary-dialogue turn sequence pairs. That outcome enables the community to start modeling and evaluation for the dialogue summarization task and we discuss initial benchmark results over this augmented set in Section~\ref{sec:benchmark}.

\subsection{Data Augmentation via Text Alignment}
We found that the summaries written by fans on the wiki are detailed, mostly ordered with respect to the corresponding episode, and mostly non-repetitive. Due to the large number of sentences in the summaries, we can break up the summaries into chunks and align each chunk to some continuous segment of the dialogue. 
\input{summary_mapping_diagram.tex}
Formally, given dialogue \(D\) consisting of \(T\) turns \(\{t_i|i\in1\dots T\}\) and summary \(S\) split into \(n\) contiguous chunks \(\{s_i|i\in1\dots n\}\), we try to determine \(A=\{a_i|i\in1\dots n\}\) where \(a_i\)   is a contiguous set of turns from \(D\) (\(a_i=t_{j:k}\)) and where \(t_j\) and \(t_k\) (\(j\leq k\)) are the earliest and latest turns in \(D\) to align  to \(s_i\); refer to Figure \ref{mapping_diagram}. To determine \(A\), we try two approaches.

\textbf{Greedy Algorithm} %For the greedy approach to this problem, w
We make an independence assumption for all \(s\) and \(t\) and try to maximize an alignment score, \(\alpha(A;S,\beta)\), where \(\beta(s,a)\) calculates an alignment score between a single \(s\) and \(a\).  
% \begin{equation}
% A=\argmax_{A,S,\beta} (\alpha)
% \end{equation}
\begin{equation}
\alpha(A;S,\beta)=\sum_{i=0}^n\max_{\substack{0\leq c \leq T \\ 0\leq w\leq 14}}(\beta(s, t_{c-w:c + w}))
\end{equation}
% \begin{equation}
% \mu(s_i, a_i)=\max_{\substack{0\leq c \leq T \\ 0\leq w\leq 14}}(\beta(s, t_{c-w:c + w}))
% \end{equation}
where bounds for \(w\) are determined empirically. For several dialogues, we tested \(0\leq w \leq T\), but this had no change in the final assignments \(A\) and greatly increased computation time. 
To choose \(\beta\), we tried several scoring functions including variations of ROUGE \cite{lin-2004-rouge}, variations of TF-IDF \cite{Jones1988ASI}, and other n-gram overlap scorings. We selected a scaled version of ROUGE-F1 score:
\begin{equation}
\begin{gathered}
\beta(s, a)=|\tau(s)\cap\tau(a)|*ROUGE_{F1} \\
={\frac{2*|\tau(s)\cap\tau(a)|^2}{|\tau(s)|+|\tau(a)|}}
\end{gathered}
\label{beta function}
\end{equation}
where \(\tau\) is a tokenization function for the given text. The scaling via \(|\tau(s)\cap\tau(a)|\) term gives extra importance to the absolute token overlap count.
\input{greedy_dp_graphs.tex}
To calculate the tokens, we found just unigrams and bigrams gave us the least noisy alignments. We also found lemmatization and stop-word removal greatly reduces the alignment quality because of the large number of n-grams (\(\geq2\)) from the turn windows that are directly used in the summaries.
% To calculate the tokens, we tried n-grams up to trigrams and found unigrams and bigrams gave us the least noisy alignments. We also experimented with lemmatization and stop-word removal but found that this greatly reduces the alignment quality because of the large number of n-grams (\(\geq2\)) from the turn windows that are directly used in the summaries.

In Figure~\ref{greedy_graph}(a), we plot the turn indices as a function of the summary chunk indices. We notice the greedy alignment approach can largely preserve the order of the summary chunks relative to the dialogue turns, without any ordering constraints.
% In Figure \ref{greedy_graph}(b), we also observe that the algorithm can mostly exclude large portions of non-narrative chit-chat from the dialogue when aligning the summary chunks, which are mostly narrative focused.
However, there are some issues with this method. First, it allows out-of-order alignments of summary chunks, which we have assessed as almost always erroneous in this dataset. Second, the recall can be low due to early cutoffs at boundaries, generally because of extensive chit-chat in between two salient utterances. 
%Trying to fix this by 
Forcing boundaries between \(a_i\) and \(a_{i+1}\) to be contiguous leads to lower precision due to salient utterances being incorrectly assigned near the borders of the turn windows. 
% Lastly, because the algorithm is greedy and maximizes over a turn window for each turn, the computation still takes a very long time. 

\textbf{Needleman-Wunsch Algorithm} The recursive approach to determining \(A\) involves imposing strict order constraints using the sequence alignment algorithm Needleman-Wunsch \cite{NEEDLEMAN1970443}, similar to \cite{nelken-shieber-2006-towards}. The algorithm imposes order by forcing \(a_i\)  and \(a_{i+1}\)  to be assigned to contiguous turn windows. We can also forgo the maximization over some window \(w\) as the algorithm does this by virtue of its score maximization function. We tried several functions for \(\beta\), including the TF-IDF function proposed by \cite{nelken-shieber-2006-towards} and found (\ref{beta function}) still performs best.
\input{nw_visualization_diagram}
To use the algorithm, we first apply \(\beta\) independently for each turn (of size 1) and summary chunk to generate a match-score matrix \(M\) of size \(T\times n\). We then build an alignment score matrix \(H\) of size \((T+1) \times (n+1)\) using:
\begin{equation}
        H_{xy}=max\left\{
                \begin{array}{ll}
                  H_{y-1,x-1}+M_{y-1,x-1}\\
                  H_{y-1,x}+M_{y-1,x-1}\\
                  H_{y,x-1}+M_{y-1,x-1}
                \end{array}
              \right.
\end{equation}
with \(M_{y-1,x-1}= \beta(s_{x-1},t_{y-1})\); \(1\leq y\leq T\); and \(1\leq x\leq n\) and the first column and row of H initialized to \(-y\) and \(-x\) respectively. We perform the traceback from \(H_{T+1,n+1}\) to \(H_{0,0}\)  to generate the alignment \(A\) where each \(a\in A\) can be seen as a vertical line in the traced path (Figure \ref{nw_vis_diagram}).

% Similar to (Nelken and Shieber 2006), 
We exclude gap penalties when generating \(H\), since we want to allow multiple turns to be assigned to a summary chunk and we want to allow a single turn to overlap several summary chunks.
% Finally, we experiment with column-wise normalization on \(M\) since we noticed that \(\beta\) can have very large ranges between columns. 
We also notice that column-wise normalization on \(M\) reduced the quality of the alignments substantially because large scores can act as an anchor for the algorithm to localize erroneous alignments. It forces the algorithm to `catch up' or `pull back' the turn alignments to include the high \(M_{y,x}\) in the final path. Normalization also reduces incentives to keep the path going down a column and heavily favors moving to the next column (summary chunk). We can visualize the improvements in Figure \ref{dp_graph}(b), where we also notice the algorithm captures turns past \(t_{1833}\) (upto \(t_{1878}\)) that were previously ignored, leading to higher recall -- we manually verified this.

The strong ordering constraint is also the source of some noise. For example, if a summary alignment overshoots the correct turn window by a large margin, it is likely that the subsequent summaries will also be misaligned due to the contiguity constraint. However, the localization effect due to large \(M\) scores help mitigate this. Another source of noise is the forced alignment of the first and last turns in dialogues that continue past the summary.

\begin{table}[!t]
\centering
\small
\begin{tabular}{lc}
\hline
\textbf{Player} & \textbf{\(\beta \)}\\
\hline
% \textbf{MATT} & 0.1133\(\pm .046\)\\
% \textbf{TRAVIS} & 0.0286\(\pm .008\)\\
% \textbf{MARIHSA} & 0.0315\(\pm .008\)\\
% \textbf{TALIESIN} & 0.0383\(\pm .011\)\\
% \textbf{SAM} & 0.0322\(\pm .010\)\\
% \textbf{ORION} & 0.0387\(\pm .014\)\\
% \textbf{LIAM} & 0.0407\(\pm .0123\)\\
% \textbf{LAURA} & 0.0298\(\pm .008\)\\
% \textbf{ASHLEY} & 0.0262\(\pm .008\)\\
\textbf{MATT} & 0.0307\(\pm .0008\)\\
\textbf{ORION} & 0.0086\(\pm .0014\)\\
\textbf{LIAM} & 0.0083\(\pm .0005\)\\
\textbf{TALIESIN} & 0.0074\(\pm .0005\)\\
\textbf{SAM} & 0.0070\(\pm .0004\)\\
\textbf{MARIHSA} & 0.0058\(\pm .0003\)\\
\textbf{TRAVIS} & 0.0057\(\pm .0004\)\\
\textbf{LAURA} & 0.0056\(\pm .0003\)\\
\textbf{ASHLEY} & 0.0048\(\pm .0006\)\\
\hline
\end{tabular}
\caption{
Mean (\(\pm\) 0.95 conf. interval) summary contribution scores for each player calculated from the normalized paths traced on \(M\) as determined by the algorithm on \(H\).
% Summary contribution scores for each player calculated from a normalized \(M\) by tracing the path on determined by the algorithm on \(H\).
}\label{tab:summary contributions}
\end{table}
We also analyze the distribution of the scores along the paths (each path normalized to 1) traced on \(M\) with respect to the nine main players (Table \ref{tab:summary contributions}). This gives us the distribution of the player contributions to the summaries.
% We also analyze of the distribution of the normalized scores of \(M\) along the path traced by the algorithm w.r.t. the 9 main players (Table \ref{tab:summary contributions}). This gives us the distribution of the player contributions to the summaries. 
% We normalize the matrix for easier comparison between players. 
Matt's turns contribute most to the summaries since he contributes the most salient narrative points. As the Dungeon Master, he is responsible for world building and the narrative's interaction with the other players. We can see the other players have much lower mean scores. One explanation for this is that they engage in more non-narrative chit chat than Matt, which leads to a lower mean \(\beta\).

\textbf{Data Augmentation} 
% The Needleman-Wunsch approach was noticeably less noisy than the greedy approach, despite some noise introduced via ordering constraint, so we proceed to data augmentation using that algorithm.
Running the Needleman-Wunsch algorithm for a dialogue \(D\) will give us \(N\) \(s,a\) pairs. We can extend this by calculating \(S\) as \(S_0\dots S_{C-1}\) where \(C\) is the chunk size and \(S_x\) is the shift in the starting point of the contiguous chunking windows. For each of these \(S_x\), we can then determine an \(A_x\) pair. This method increases our \(s, a\) pairs by a factor of \(C\). We can go further by running this for different chunk sizes. For our experiment, we chose to run this algorithm for \(C\)=2, 3, and 4 sentences. We remove dialogues with \(|S|\leq10\) chunks  (since there are some incomplete wikis) and get 55385 \(s, a\) pairs. To reduce noise, we also: (1)~impose \(2<|t_{j:k}|\leq100\); and (2)~strip out pairs where \(s_i\) contains ``Q: " (signifies a differently formatted question answer segment in an episode). We end up with 34243 pairs (Table \ref{tab:data augmentation}), a substantial increase from the original 159 summary, dialogue pairs. Refer to Figure \ref{sample-dialogue} and to the Appendix for examples of the summaries and examples. These are then split as 26232 training, 3470 validation, and 4541 testing \(s, a\) pairs; refer to Appendix for details.
\begin{table}[!t]
\centering
\small
\begin{tabular}{ccc}
\hline
\textbf{Chunk Size} & \textbf{w/o Filtering} & \textbf{w/ Filtering}\\
\hline
2 & 18569 & 11124\\
3 & 18438 & 11635\\
4 & 18378 & 11484\\
\hline
\end{tabular}
\caption{number of \(s_i, a_i\) pairs generated for each chunk size with and without filtering.}\label{tab:data augmentation}
\end{table}

We calculate precision and recall with respect to the turns on a random sample of 100 pairs from the training split of these 34243 pairs and obtain a precision of 0.8692 and recall of 0.9042. Refer to Appendix for precision and recall calculation method. We find precision errors are mostly from extraneous trailing or leading turns attached to the properly aligned set of turns, and almost never from complete misalignment.
% We find precision errors are mostly from extraneous turns being included in the turn window, but almost never from a completely misaligned turn window. In the worst examples, there are extremely long trailing or leading turn sequences attached to the properly aligned section of the turn window. 
We find recall errors are from turn sequences that start too late or end too early, and also almost never from complete misalignment.
% Like precision errors, we find recall errors are almost never from a completely misaligned turn window. Rather, they are from turn sequences starting too late or ending too early. 
In most cases where \(a\)  contains a recall error, we notice the precision for that \(a\) is 1.0, because \(a\) ends up being a subset of the correct \(t_{j:k}\). We posit this is due to the strong order constraints of the algorithm and our post-alignment filtering, which removes the pairs with the highest risk of complete misalignment.

As a measure of quality of the human written summaries, we also perform a question-answering task on a random sample of 50 \(s_i, a_i\) pairs from the filtered set. First the questioner records two questions and answers per pair, with the questions and answers coming only from the summaries \(s_i\). For each pair, there is one factoid question with an open-ended answer and one multiple choice question with four possible answers. The factoid question can be answered by yes|no responses, entity names, or short text. The multiple choice question has at most one correct answer of the four contained in the summary chunks.  (Figure \ref{fig: sample questions}). The questions are then answered by another person, using only the aligned turns \(a_i\) from the pair.

\begin{figure}[!t]
\centering
\small
\begin{tabular}{p{6.6cm}}
\hline
\textbf{Summary} \\
\hline
``The Mighty Nein make their way up the ladder and through the hatch into the Keystone Pub proper, where they order breakfast. A hooded female wearing a long green cloak covering her left face and side approaches and asks if they're heading into the swamp today-- she's desperate to go there herself. Calianna apologizes for bothering them, but she couldn't help but overhear their conversation last night."\\
\hline
\textbf{Factoid Question} \\
\hline
1. Who was overhearing the Mighty Nein's conversation the previous night?\\
\hline
\textbf{Multiple Choice Question} \\
\hline
2. What do the Mighty Nein have at the Keystone Pub? \\
\textbf{(A)} drinks \textbf{(B)} dinner \textbf{(C)} lunch \textbf{(D)} breakfast\\
\hline
\end{tabular}
\caption{Example of questions constructed for a human-written summary chunk aligned to a set of turns.}\label{fig: sample questions}
\end{figure}

\begin{table}[!t]
\centering
\small
\begin{tabular}{lccc}
\hline
\textbf{Question Type} & \textbf{Correct} & \textbf{Incorrect} & \textbf{Precision}\\
\hline
Free Form & 39 & 11 & 78\%\\
Multiple Choice & 42 & 8 & 84\%\\
Total & 81 & 19 & 81\%\\
\hline
\end{tabular}
\caption{Correct and incorrect answers for the Q\&A evaluation method, for measuring precision w.r.t. the human written summaries in the \(s_i, a_i\) pairs.}\label{tab:qa evaluation table}
\end{table}

The scores are recorded in Table \ref{tab:qa evaluation table}. Out of the 19 incorrect answers, we found that 17 of them were due to summary alignment errors. This is where the correct information was in the dialogue, but not in the aligned set of turns. The other 2 were due to misinterpretation of the question when answering. This indicates, with perfect alignment, all questions could have been answered correctly; meaning what is in the summaries is an accurate reflection of what is in the transcript. However, we recognize all the information in the transcripts is not necessarily in the summaries; for example, out-of-game information. We also notice that multiple choice questions have a higher accuracy due to easier questions and additional context provided by the set of answers themselves, and not due to random guessing. We also found that 12 incorrect answers were due to no answer, meaning the answerer did not feel they had enough information to attempt an answer. For the other 7, the answerer felt that at least some information pertaining to the question was available in the aligned turns.

Unlike ROUGE precision, which relies on word overlap, this evaluation can incorporate latent semantic and contextual information. It is important to note that latent information used when answering varies greatly between people, making this method subjective with respect to the answerer. In future work, it would be interesting to measure variance of accuracy and information in the answers using a large number of people.





