 % This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage[]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{listings}% http://ctan.org/pkg/listings
\lstset{
  basicstyle=\ttfamily,
  mathescape
}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage[subtle]{savetrees}
\usepackage{microtype}
\makeatletter
\MT@redefine@patch{eqnum}{%
  \MT@patch@patch\tagform@{(}{\leftprotrusion{(}}%
  \MT@patch@patch\tagform@{)}{\rightprotrusion{)}}%
  \MT@ifdefined@n@TF{eqref }
    {\MT@exp@cs\MT@patch@patch{eqref }}{\MT@patch@patch\eqref}
       {\tagform@}{\@nameuse{MT@patch@saved@\string\tagform@}}%
}{}
\makeatother

\usepackage{url}
\usepackage{times}
\usepackage{latexsym}
\renewcommand{\UrlFont}{\ttfamily\small}
\usepackage{multicol}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage{verbatim}
\usepackage{makecell}
\usepackage{nicefrac, xfrac}
\usepackage{braket}
\usepackage{hhline}
\usepackage{amsmath,bm,amssymb}
\usepackage[linesnumbered,ruled]{algorithm2e}
\usepackage{xcolor}
\definecolor{Green}{rgb}{0.0, 0.5, 0.0}
\definecolor{Amethyst}{rgb}{0.6, 0.4, 0.8}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{\texttt{Chain of Density}: Learning an Optimal Level of Detail \\ by Iteratively Fusing Entities into a Fixed Length Summary}

\author{
  Griffin Adams$^{\spadesuit,\clubsuit}$ \\ griffin.adams@columbia.edu \And
  Alexander R. Fabbri$^{\diamondsuit}$ \\ afabbri@salesforce.com \\ \AND Faisal Ladhak $^{\spadesuit}$ \\ faisal@cs.columbia.edu \\\And No\'emie Elhadad$^{\spadesuit,\clubsuit}$ \\ noemie.elhadad@columbia.edu \AND
 Salesforce Research$^{\diamondsuit}$ \quad  Columbia University: Computer Science$^{\spadesuit}$, Biomedical Informatics$^{\clubsuit}$
}

\begin{document}
\maketitle
\begin{abstract}

Selecting the ``right'' amount of information to include in a summary is a difficult task. A good summary should be detailed and entity-centric without being overly dense and hard to follow. To better understand this tradeoff, we solicit increasingly dense GPT-4 summaries with a "Chain of Density" prompt. Specifically, GPT-4 generates an initial entity-sparse summary before iteratively incorporating missing salient entities without increasing the length. We conduct a human preference study on 100 CNN DailyMail articles and discover that humans prefer GPT-4 summaries which are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Based on these preferences, we compute an optimal fixed number of entities to include in each summary and generate distillation training data with GPT-4.


-length LLAMA2 summarization model.

\end{abstract}

\section{Introduction}

Automatic summarization has come along way in the past few years, largely due to the strong zero-shot capabilities of Large Language Models (LLMs). Additionally, careful prompting can enable fine-grained control over summary characteristics, such as length \citep{goyal2022news}, topics \citep{bhaskar-etal-2023-prompted}, and style \citep{pu-demberg-2023-chatgpt}.

Yet, an overlooked aspect is the information density of an summary. In theory, as a compression of another text, a summary \emph{should} be denser and contain more entities per token than the source document. Yet, how dense is an open question. A summary is uninformative if it contains insufficient detail. Yet, if it contains too much information, it can become difficult to follow without having to increase the overall length. Conveying more information subject to a fixed token budget requires a combination of abstraction, compression, and fusion. There is a limit to how much space can be made for additional facts before a text becomes illegible or even factually incorrect.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images/Density-Plot.pdf}
\caption{Summaries produced with \texttt{Chain of Density} prompting grow increasingly dense, starting off closer to vanilla GPT-4 summaries and eventually surpassing that of human written summaries. We generate summaries with variable densities to determine an optimal level.}
\label{fig:density}
\end{figure}

In this paper, we seek to identify this limit by soliciting human preferences on a set of increasingly dense summaries produced by GPT-4. Specifically, we start off by generating an entity-sparse summary. Then, we iteratively identify and fuse 1-3 missing entities from the previous summary without increasing the overall length of the summary ($4x$). Each summary has a higher ratio of entities to tokens than the previous. Based on human preference data, we determine that humans prefer summaries which are almost as dense as human-written summaries and more dense than those generated by a vanilla GPT-4 prompt.

Given a document to summarize and a fixed token budget, we compute the optimal number of entities to include and prompt GPT-4 to identify them before incorporating them into a summary. We then distill this entity-dense, chain-of-thought style prompt into an open source model (LLAMA-2) and show that it produces more relevant 

Our primary contributions are: X Y Z.

\section{Chain of Density Prompting} \label{sec:prompt}

\paragraph{Prompt.} Our goal is to generate a set of summaries with GPT-4 with varying levels of information density, while controlling for other confounding variables such as length. To do this, we formulate a single \texttt{Chain of Density} (\textbf{\texttt{CoD}}) prompt, whereby an initial summary is generated and made increasingly entity dense. Specifically, for a fixed number of turns, a set of unique salient entities from the source text are identified and fused into the previous summary without increasing the length. The first summary is entity-sparse as it focuses on only 1-3 initial entities. To maintain the same length while increasing the number of entities covered, abstraction, fusion, and compression is explicitly encouraged. Specifically, \textbf{\texttt{CoD}} is solicited with the following prompt:

\begin{verbatim}
Article: {{ARTICLE}}

You will generate increasingly
concise, entity-dense summaries
of the above Article.

Repeat the following 2 steps
5 times.

Step 1. Identify 1-3 informative
Entities (";" delimited) from the
Article which are missing from
the previously generated summary.

Step 2. Write a new, denser
summary of identical length which
covers every entity and detail
from the previous summary plus
the Missing Entities.

{{Additional Guidelines}}
\end{verbatim}

Please refer to Appendix \ref{app:prompts} for the Additional Guidelines provided to GPT-4, which promote concise, yet lossless, densification of the summaries.

\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{images/Density-Statistics.png}
\caption{Summaries produced with \texttt{Chain of Density} prompting grow increasingly dense xxx.}
\label{fig:implicit-stats}
\end{figure*}

\paragraph{Data.} We randomly sample 100 articles from the CNN/DailyMail summarization dataset for which to generate Chain-of-Density summaries.

\paragraph{Statistics.} We bifurcate statistics into explicit and implicit. Explicit statistics (tokens, entities, entity density) are ones directly controlled for by \texttt{CoD}, while the implicit statistics are byproducts of densification (abstractiveness, fusion, content distribution). Abstractiveness should increase with each CoD step because summaries are iteratively re-written to make space for each additional entity. We measure abstractiveness with extractive density: the average squared length of extractive fragments (grusky todo). Similarly, the level of concept fusion should increase monotonically as entities are added to a fixed-length summary. We proxy fusion as average number of source sentences aligned to each summary sentence. For alignment, we use the relative ROUGE gain method from Neusum, which aligns source sentences to a target sentence until the relative ROUGE gain of an additional sentence is no longer positive. We would expect \texttt{CoD} summaries to initially exhibit a strong Lead Bias yet gradually start to pull in entities from the middle and end of the article. To measure this, we use our alignments from fusion and measure the average sentence rank of aligned source sentences (averaged across each summary sentence).


% Lastly, particularly for news articles, salient concepts are disproportionately drawn from the beginning of the article, a phenomenon known as the Lead Bias. 

We compare Chain-of-Density summaries to the human-written bullet-point style references as well as summaries generated by GPT-4 with a vanilla prompt: 

\begin{verbatim}
Article: {{ARTICLE}}

Summarize the above Article in
70 words.
\end{verbatim}

Figure \ref{fig:implicit-stats} shows XYZ.

\section{Results} \label{sec:results}


\begin{table*}[t]
\centering
\small
\begin{tabular}{cc|ccccc|c}
\textbf{CoD Step} & \textbf{Entity Density} & \textbf{Informative} & \textbf{Quality} & \textbf{Coherence} & \textbf{Attributable} & \textbf{Overall} & \textbf{GPT-4 Eval Average} \\ \hline
\textbf{1} & 0.089 & 2.77 & 2.48 & 2.81 & 4.03 & 2.51 & 2.92 \\
\textbf{2} & 0.129 & 3.71 & 3.77 & 3.89 & 4.60 & 3.57 & 3.91 \\
\textbf{3} & 0.148 & 4.30 & \textbf{4.26} & \textbf{4.42} & \textbf{4.86} & 4.13 & \textbf{4.39} \\
\textbf{4} & 0.158 & 4.52 & 4.06 & 4.14 & 4.64 & \textbf{4.19} & 4.31 \\
\textbf{5} & 0.167 & \textbf{4.69} & 3.36 & 3.58 & 4.51 & 2.88 & 3.80 \\
\hline
\end{tabular}
\caption{GPT-4 Likert-scale (1-5) assessments of Chain of Density Summaries by step. xx } \label{tab:gpt4-likert}
\end{table*}

\paragraph{Automatic Metrics.}

Table \ref{tab:gpt4-likert}.

\begin{table}[t]
\centering
\small
\begin{tabular}{c|ccccc}
& \multicolumn{5}{c}{\textbf{\% Share of First Place Votes}} \\
\textbf{CoD Step} & \multicolumn{4}{c}{\textbf{Individual}} & \textbf{Aggregate} \\ \hline
\textbf{1} & 3.0 & 2.0 & 13.0 & 17.4 & 8.3 \\
\textbf{2} & 25.0 & \textbf{28.0} & \textbf{43.0} & \textbf{31.4} & \textbf{30.8} \\
\textbf{3} & 22.0 & \textbf{28.0} & 21.0 & 24.4 & 23.0 \\
\textbf{4} & \textbf{29.0} & 25.0 & 13.0 & 26.7 & 22.5 \\
\textbf{5} & 21.0 & 17.0 & 10.0 & 16.3 & 15.5 \\
\hline
\end{tabular}
\caption{Human Preferences. } \label{tab:human-preferences}
\end{table}

\paragraph{Human Preferences.}

Table \ref{tab:human-preferences}.


\section{Conclusion.}

% We open-source LLAMA-2 \texttt{IncrSumm}, a model which learns to generate dynamic length summaries of \textit{just the right length} without any hand-crafted prompts. We distill the knowledge from a more expensive procedure (lengthen-then-truncate with GPT-4) into a simpler LLAMA-2 model with just 1,000 CNN/Dailymail articles. We demonstrate that \texttt{IncrSumm} summaries are xxx.

% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}

\appendix

\section{Prompts} \label{app:prompts}

Below, we show the prompt for the Additional Guidelines for Fusing and Adding content.

\begin{verbatim}
A Missing Entity is:
- Relevant: to the main story.
- Specific: descriptive yet
 concise.
- Novel: not in the previous
  summary.
- Faithful: present in the
  Article.
- Anywhere: located anywhere
  in the Article.

Guidelines:
- The first summary should be long
(4-5 sentences, ~80 words) yet highly
non-specific, containing little
information beyond the entities
marked as missing. Use overly
verbose language and fillers
(e.g., "this article discusses")
to reach ~80 words.
- Make every word count: re-write the
previous summary to improve flow and
make space for additional entities.
- Make space with fusion, compression,
and removal of uninformative phrases
like "the article discusses".
- The summaries should become highly dense
and concise yet self-contained, e.g.,
easily understood without the Article.
- Missing entities can appear anywhere
in the new summary.
- Never drop entities from the previous
summary. If space cannot be made,
add fewer new entities.

Remember, use the exact same number
of words for each summary.

Answer in JSON. The JSON should be a
list (length 5) of dictionaries whose
keys are "Missing_Entities" and
"Denser_Summary".
\end{verbatim}


\end{document}
