@inproceedings{cao-etal-2025-multi2,
    title = "Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing",
    author = "Cao, Juntai  and
      Zhang, Xiang  and
      Li, Raymond  and
      Wei, Jiaqi  and
      Li, Chuyuan  and
      Joty, Shafiq  and
      Carenini, Giuseppe",
    editor = "Dong, Yue  and
      Xiao, Wen  and
      Zhang, Haopeng  and
      Zhang, Rui  and
      Ernst, Ori  and
      Wang, Lu  and
      Liu, Fei",
    booktitle = "Proceedings of The 5th New Frontiers in Summarization Workshop",
    month = nov,
    year = "2025",
    address = "Hybrid",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.newsum-main.10/",
    pages = "135--156",
    ISBN = "979-8-89176-337-1",
    abstract = "Recent advances in test-time scaling have shown promising results in improving large language model performance through strategic computation allocation during inference. While this approach has demonstrated strong improvements in reasoning tasks, its application to natural language generation tasks, particularly summarization, remains unexplored.Among all of the generation tasks, multi-document summarization (MDS) presents unique challenges by requiring models to extract and synthesize essential information across multiple lengthy documents. Unlike reasoning tasks, MDS demands a more complicated approach to prompt design and ensemble methods, as no single ``best-overall'' prompt can satisfy diverse summarization requirements. The inherent diversity in summarization needs necessitates exploring how different prompting strategies can be systematically combined to improve performance.We propose a novel framework that harnesses prompt diversity to enhance MDS performance. Our approach generates multiple candidate summaries using carefully designed prompt variations, then ensemble them through sophisticated aggregation methods to produce refined summaries. This prompt diversity enables models to capture different aspects and perspectives of the source documents, leading to more comprehensive and higher-quality summaries. To evaluate our method effectively, we also introduce two new LLM-based metrics: the Preference Alignment Score (PAS) and LLM Atom-Content-Unit score (LLM-ACU), which assess summary quality while addressing the positional bias inherent in automatic evaluations performed by LLMs.Our experiments demonstrate that leveraging prompt diversity significantly enhances summary quality, while also revealing the practical scaling boundaries for MDS tasks."
}