@inproceedings{zhang-etal-2026-deepsynth,
    title = "{D}eep{S}ynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing",
    author = "Zhang, Hongzhi  and
      Hu, Yuanze  and
      Zhang, Tinghai  and
      Fu, Jia  and
      Wang, Tao  and
      Jing, Junwei  and
      Fan, Zhaoxin  and
      Bi, Wei  and
      Tang, Ruiming  and
      Li, Han  and
      Zhou, Guorui  and
      Gai, Kun",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1688/",
    pages = "33805--33822",
    ISBN = "979-8-89176-395-1",
    abstract = "The evolution of Large Language Models (LLMs) towards autonomous agents has catalyzed progress in Deep Research. While retrieval capabilities are well-benchmarked, the post-retrieval synthesis stage{---}where agents must digest massive amounts of context and consolidate fragmented evidence into coherent, long-form reports{---}remains under-evaluated due to the subjectivity of open-ended writing.To bridge this gap, we introduce DeepSynth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities. We leverage high-quality survey papers as gold standards, reverse-engineer research requests, and construct Oracle Contexts from their bibliographies to isolate synthesis from retrieval noise. We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization), transforming subjective judgment into verifiable metrics. Experiments across 96 tasks reveal that synthesizing information from hundreds of references remains a significant challenge. Our results demonstrate that agentic ``plan-then-write'' workflows significantly outperform single-turn generation, effectively reducing hallucinations and improving adherence to complex structural constraints."
}