% \subsection{Data Format}
\subsection{Summary Dialogue Alignment Precision and Recall Calculation Method}
We calculate precision and recall for summary dialogue alignment with respect to the dialogue's turns in Section 4.1. Here, we describe our method for calculating precision and recall.

Precision is expressed as a function of true positives and false positives and recall is expressed as a function of true positives and false negatives. For each alignment \(a_i \in A\), we classify each of its turns \(t\) as a True Positive (TP), False Positive (FP), or False Negative (FN).
We take the counts of all TP, FP, and FN over the entire \(A\) and perform the precision and recall calculations, precision=\(\frac{total(TP)}{total(TP)+total(FP)}\), recall=\(\frac{total(TP)}{total(TP)+total(FN)}\).
 
\subsubsection{TP, FP, FN Classifications} We have the following guidelines to classify a turn in \(a_i\) as a TP, FP, or FN.

\begin{enumerate}
  \item First, find the earliest and latest turns in the original dialogue that correspond to the summary chunk \(s_i\).  All alignments \(a \in A\) are a contiguous sequence of turns extracted from the dialogue. For example, in the summary chunk in Figure \ref{sample-dialogue}, the earliest turn in the entire dialogue that corresponds to the summary is (1) in the alignment. The latest turn in the entire dialogue that corresponds to the summary is (7) in the alignment (we verify this by looking at the turns in the original dialogue before and after the sequence presented in the alignment).
  
  \item Any turn in the alignment in between the earliest and latest turns identified in Step 1 (inclusive) is considered a true positive. Any turn in the alignment outside of the earliest and latest turns identified in Step 1 is considered a false positive. In Figure \ref{sample-dialogue}, turn (0) would be considered a false positive because it does not correspond to any of the summary sentences (0,1,2,3). Turns (1,2,3,4,5,6,7) are considered true positives since they are between the earliest and latest turns that correspond to the summary sentences in original dialogue.
  
  \item Any turn between the earliest and latest turns identified in Step 1 that is NOT present in the alignment is considered a false negative. In Figure \ref{sample-dialogue}, if the turn (7) was not in the alignment, it would be considered a false negative because the turn (7) corresponds to the summary sentence (2) and is between the earliest and latest turns identified in Step 1 (turns 1 and 7 respectively).
\end{enumerate}
% \input{sample_dialogue_table.tex}
\subsection{More Examples of Summary-Dialogue Alignments}
We give more examples of summary-dialogue alignments (\(s_i, a_i\)) pairs. For the sake of brevity, we chose to show examples that were only 10 turns or smaller. Please refer to the dataset itself for much longer samples.

In Figure \ref{sample-dialogue-recall-error}, we have an alignment with a large recall error. 
\input{sample_dialogue_table_2} 
In Figure \ref{sample-dialogue-out-of-game}, we have an example of a summary referring to out-of-game turns. We find these types of summaries are typically written for break-times in the show, before the start of a game session, or after the end of a game session. Generally, they seem to make up a smaller portion of the overal summary content. This example in particular is for a Q/A session the team held after their session\footnote{\href{https://criticalrole.fandom.com/wiki/Attack_on_the_Duergar_Warcamp}{Attack\_on\_the\_Duergar\_Warcamp episode}}.
\input{sample_dialogue_table_3}
In Figure \ref{sample-dialogue-latent}, we have a perfect alignment, with the summary explicitly capturing implied information in the turns. There are also examples of role-playing by Matt in this turn sequence, as he speaks to the other players from the perspective of the in-game character Ripley. This is shown through the use of quotes in turns 0, 4, and 6.
\input{sample_dialogue_table_4}

\subsection{Train, Validation, Test Split Method}
In Section 4.1, we split the aligned 34243 pairs into 26232 training, 3470
validation, and 4541 testing pairs. Here, we briefly describe our method.

We first split the 159 dialogues into an (80\%, 10\%, 10\%) train, validation, and test split based on order. This guarantees that episodes from validation will succeed episodes in training, and episodes in testing will succeed episodes in validation. We take all the \(s, a\) pairs from these dialogues and put them into their respective train, validation, test sets. We chose to split by this method so that (1) there will never be an episode that is in more than one train/val/test set; (2) no summary of chunk size \(C_i\) from validation or testing is a subset of summary of chunk size \(C_j\) from the training set where \(i\leq j\), thus avoiding bias in the final metrics; and (3) we can train on information that happened in the show prior to information we validate or test on, thus better mimicking a real-world scenario where you cannot train on future information.

As new Critical Role episodes and seasons are added, we hope to expand the CRD3 dataset correspondingly. Future work might include splitting the training, validation, and testing sets based on season or some method that guarantees independence between narrative elements from the summaries and turns in the training, validation, and testing sets. Note, as new Critical Role episodes are added, we will keep the original version preserved so as to keep the experiments and analysis reproducible.