@inproceedings{chen-etal-2026-peercheck,
    title = "$\texttt{PeerCheck}$: Enhancing {LLM}-Generated Academic Reviews Towards Human-Level Quality",
    author = "Chen, Zeyuan  and
      Yang, Ziqing  and
      Ma, Yihan  and
      Backes, Michael  and
      Zhang, Yang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1170/",
    pages = "23362--23386",
    ISBN = "979-8-89176-395-1",
    abstract = "As academic submissions grow, the traditional peer review process struggles to keep up, raising concerns about quality and fairness.A trend of using large language models (LLMs) for assistance has emerged.In this work, we take a critical step toward improving the quality of LLM-generated reviews.We propose the $\texttt{PeerCheck}$ framework, which investigates LLM-human review differences ($\textbf{RQ1}$) and explores methods to increase LLM-human similarity ($\textbf{RQ2}$).We first analyzed the human-written reviews with reviews generated by GPT-4o, Claude-3.7-Sonnet, and DeepSeek-V3 and found that LLMs and humans focus on different terms, e.g., LLMs prioritize theory while humans emphasize methodology and experiments.We further adopt prompt engineering, such as Chain-of-Thought (CoT), and utilize retrieval-augmented generation (RAG) to enhance the LLM-generated reviews towards human-level quality.We find CoT significantly improves the human similarity of LLM reviews, while we also discover an unexpected ``RAG paradox,'' i.e., experiments with RAG produce different results for various LLMs and, in some cases, even reduce review quality.Our comprehensive analysis of LLM-generated academic reviews illustrates both possibilities and limitations, contributing to a more effective, human-aligned review system."
}