@inproceedings{kim-etal-2026-scaling,
    title = "Scaling Evaluation-Time Compute with Reasoning Models as Evaluators",
    author = "Kim, Seungone  and
      Wu, Ian  and
      Lee, Jinu  and
      Yue, Xiang  and
      Lee, Seongyun  and
      Moon, Minkyeong  and
      Lawrence, Carolin  and
      Gashteovski, Kiril  and
      Hockenmaier, Julia  and
      Neubig, Graham  and
      Welleck, Sean",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2102/",
    pages = "42354--42384",
    ISBN = "979-8-89176-395-1",
    abstract = "Language model (LM) evaluators that generate chain-of-thought (CoT) reasoning are widely used for the assessment of LM responses. Simultaneously, increasing LMs' ``thinking'' time through scaling test-time compute has proven to be an effective technique for solving challenging problems in domains such as math and code. This raises a natural question: can an LM{'}s evaluation capability also be improved by scaling test-time compute? To answer this, we investigate employing reasoning models - LMs that natively generate long CoT reasoning - as evaluators. We explore scaling evaluation-time compute by using reasoning models to evaluate both the overall candidate response (i.e., outcome evaluation) and the individual reasoning steps within it (i.e., process evaluation). We observe that evaluator performance improves monotonically with the number of reasoning tokens generated, mirroring trends seen in LM reasoning. Furthermore, we use these more accurate evaluators to rerank multiple generations, and demonstrate that spending more compute at evaluation time can be as effective as increasing compute during generation for improving an LM{'}s problem-solving performance."
}