@inproceedings{wu-etal-2026-novbench,
    title = "{N}ov{B}ench: Evaluating Large Language Models on Academic Paper Novelty Assessment",
    author = "Wu, Wenqing  and
      Zhao, Yi  and
      Wang, Yuzhuo  and
      Li, Siyou  and
      Shao, Juexi  and
      Long, Yunfei  and
      Zhang, Chengzhi",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1607/",
    pages = "32103--32133",
    ISBN = "979-8-89176-395-1",
    abstract = "Novelty is a core requirement in academic publishing and a central focus of peer review, yet the growing volume of submissions has placed increasing pressure on human reviewers. While large language models (LLMs), including those fine-tuned on peer review data, have shown promising results in generating review comments, the absence of dedicated benchmarks has limited systematic evaluation of their ability to assess research novelty. To address this gap, we introduce NovBench, the first large-scale benchmark designed to evaluate LLMs' capability to generate novelty evaluations in support of human peer review. NovBench comprises 1,684 paper{--}review pairs from a leading NLP conference, including novelty descriptions extracted from paper introductions and corresponding expert-written novelty evaluations. We focus on both sources because the introduction provides a standardized and explicit articulation of novelty claims, while expert-written novelty evaluations constitute one of the current gold standards of human judgment. Furthermore, we propose a four-dimensional evaluation framework (including Relevance, Correctness, Coverage, and Clarity) to assess the quality of LLM-generated novelty evaluations. Extensive experiments on both general and specialized LLMs under different prompting strategies reveal that current models exhibit limited understanding of scientific novelty, and that fine-tuned models often suffer from instruction-following deficiencies. Our findings underscore the need for targeted fine-tuning strategies that jointly improve novelty comprehension and instruction adherence."
}