@inproceedings{pham-etal-2026-analystbench,
    title = "{A}nalyst{B}ench: Benchmarking professional long-form report generation with web-mined multimodal tasks",
    author = "Pham, Chau Minh  and
      Wang, Zichao  and
      Mathur, Puneet  and
      Siu, Alexa  and
      Jain, Akriti  and
      Garimella, Aparna  and
      Sai, Ananya B.  and
      Lipka, Nedim  and
      Iyyer, Mohit  and
      Manjunatha, Varun",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1197/",
    pages = "23894--23926",
    ISBN = "979-8-89176-395-1",
    abstract = "Large language models are increasingly used to draft long-form multimodal documents, but their end-to-end performance on professional report generation remains systematically understudied. We introduce AnalystBench, a continually extensible benchmark of 20 real-world report generation tasks grounded in multimodal document collections, where models must process millions of input tokens to produce long-form professional reports. Using expert-validated quality checklists and groundedness evaluation, we evaluate LLMs and coding agents and find that the best model, GPT-5.1, scores highly on executive summarization tasks (exceeding 90{\%} on quality checklists) but degrades substantially on tasks requiring long-horizon synthesis over large inputs (dropping to 25-40{\%}). Agent-based generation substantially benefits strong closed-source models such as GPT-5.1, with checklist scores improving by 20.24 percentage points and visual coverage by 37.41 points over vanilla generation, but offers little or negative gains for open-source models like DeepSeek-R1 (-3.02 points). Expert reviewers note that while generated reports are grounded and clearly separate factual description from interpretation, they often fall short in actionability, clarity, and quantitative precision, which highlights the gap between system performance and real-world professional needs."
}