@inproceedings{yang-etal-2026-changjuan,
    title = "{C}hang{J}uan: A Comprehensive Benchmark for Book-Length {C}hinese Story Evaluation",
    author = "Yang, Dingyi  and
      Wang, Mingshuo  and
      Jin, Qin",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2044/",
    pages = "41116--41134",
    ISBN = "979-8-89176-395-1",
    abstract = "Automatic evaluation of book-length stories remains underexplored, particularly for non-English literature. We introduce ChangJuan, the first benchmark for *book-length Chinese story evaluation*, comprising 300 novels with metadata, human ratings, and large-scale user reviews. To mitigate the subjectivity of raw reviews, we propose a distillation method to aggregate them into generally agreed viewpoints (pros and cons) across key evaluation aspects such as plot and character. We conduct systematic experiments to benchmark current LLMs, analyze aspect importance, and examine genre differences. For book-length story evaluation, we propose an enhanced summary-based method that leverages length-detail balanced summaries and representative excerpts, generates aspect-specific reviews, and considers genre-aware aspect weighting to assign a final score. Using this framework and our distilled viewpoints, we fine-tune an 8B model, CLEM, which outperforms open-source baselines and raises Qwen3{'}s Kendall{'}s tau correlation with human judgments from 24.8 to 34.1. Our datasets and codes are available at https://github.com/DingyiYang/ChangJuan."
}