@inproceedings{zeng-etal-2026-tinyjudge,
    title = "{T}iny{J}udge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles",
    author = "Zeng, Yirong  and
      Liu, Yufei  and
      Ding, Xiao  and
      Hou, Yutai  and
      Wang, Yuxian  and
      Ning, Wu  and
      Song, Haonan  and
      Tu, Dandan  and
      Zhang, Qixun  and
      He, Yuxiang  and
      Cai, Bibo  and
      Liu, Ting",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1204/",
    pages = "26198--26212",
    ISBN = "979-8-89176-390-6",
    abstract = "Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints, ranging from verifiable ones (e.g., output length) to unverifiable ones (e.g., tone). Reinforcement learning with verifiable rewards has emerged as a paradigm for IF tasks, leveraging LLM-as-a-judge to assess unverifiable constraints. However, we empirically find that this approach remains a significant bottleneck, suffering from severe reward hacking and higher computational overhead. In this work, we first analyze the generalization capabilities of unverifiable constraints and discover that specific constraints exhibit distinct, high-generalization patterns. Motivated by this, we propose TinyJudge, a framework that employs an ensemble of specialized tiny language models (e.g., 0.6B) to provide rewards for soft constraints. By distilling expertise from frontier models into these tiny models, it achieves high-precision, lightweight evaluation. Extensive evaluations across five benchmarks demonstrate that TinyJudge outperforms the baselines by {\textasciitilde}10{\%} in average performance and 12{\%} in reward precision. Crucially, it also achieves a 3{\texttimes} speedup in total training time. Our work provides a scalable and robust path for aligning LLMs with unverifiable human instructions."
}