@inproceedings{lee-etal-2026-efficient,
    title = "Efficient Process Reward Modeling via Contrastive Mutual Information",
    author = "Lee, Nakyung  and
      Hong, Sangwoo  and
      Lee, Jungwoo",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1744/",
    pages = "37603--37617",
    ISBN = "979-8-89176-390-6",
    abstract = "Recent research has devoted considerable effort to verifying the intermediate reasoning steps of chain-of-thought (CoT) trajectories using process reward models (PRMs) and other verifier models. However, training a PRM typically requires human annotators to assign reward scores to each reasoning step, which is both costly and time-consuming. Existing automated approaches, such as Monte Carlo (MC) estimation, also demand substantial computational resources due to repeated LLM rollouts. To overcome these limitations, we propose **contrastive pointwise mutual information (CPMI)**, a novel automatic reward labeling method that leverages the model{'}s internal probability to infer step-level supervision while significantly reducing the computational burden of annotating dataset. CPMI quantifies how much a reasoning step increases the **mutual information** between the step and the correct target answer relative to hard-negative alternatives. This **contrastive** signal serves as a proxy for the step{'}s contribution to the final solution and yields a reliable reward. The experimental results show that CPMI-based labeling reduces dataset construction time by **84{\%}** and token generation by **98{\%}** compared to MC estimation, while achieving higher accuracy on process-level evaluations and mathematical reasoning benchmarks."
}