@inproceedings{fein-etal-2026-litbench,
    title = "{L}it{B}ench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing",
    author = "Fein, Daniel  and
      Russo, Sebastian  and
      Xiang, Violet  and
      Jolly, Kabir  and
      Rafailov, Rafael  and
      Haber, Nick",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.362/",
    pages = "7740--7755",
    ISBN = "979-8-89176-380-7",
    abstract = "Evaluating creative writing generated by large language models (LLMs) remains challenging because open-ended narratives lack ground truths. Without performant automated evaluation methods, off-the-shelf (OTS) language models are employed as zero-shot judges, yet their reliability is unclear in this context. To address this gap, we introduce LitBench, a large-scale benchmark for creative writing evaluation, featuring a training corpus of 43,827 story pairs and a 2,480-pair test set curated from Reddit. Using LitBench, we benchmark existing LLM judges and train specialized reward models. Our analysis reveals that the strongest OTS judge, Claude-3.7-Sonnet, achieves only 73{\%} agreement with human preferences. In contrast, our trained Bradley-Terry and generative reward models both reach 78{\%} accuracy, outperforming all OTS judges. An online human study further validates our models, showing their rankings of newly generated stories align more closely with human preferences. Our work provides the first reliable benchmark and specialized reward models for creative writing, establishing a crucial foundation for the future development of more capable verifiers."
}