@inproceedings{kola-etal-2026-saja,
    title = "{SAJA}: A Simple Approach to Judge Alignment for {LLM}-as-a-Judge",
    author = "Kola, Sneha  and
      Sharma, Pankaj Kumar  and
      Dey, Soumyadeep  and
      Mishra, Bamdev  and
      Datar, Mayur",
    editor = "Li, Yunyao  and
      Rehm, Georg  and
      Tu, Mei",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics ({ACL} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-industry.45/",
    pages = "646--664",
    ISBN = "979-8-89176-394-4",
    abstract = "LLM-as-a-Judge systems are increasingly used to evaluate text at scale, yet production deployment demands low latency, minimal cost, and compatibility with closed-source APIs. Current approaches fall short in different ways: some require many LLM calls and per-dataset prompt tuning, others depend on logit access unavailable in commercial APIs, and yet others demand multiple rounds of LLM interaction for iterative feature discovery. We present **SAJA** (**S**imple **A**pproach to **J**udge **A**lignment), built on the principle that task-specific alignment should reside in a lightweight calibration head, not in elaborate prompts or model internals. SAJA makes exactly one LLM call per item using a fixed structured rubric prompt, extracts a multi-dimensional feature vector, and maps it to a human-aligned score via a calibration head trained on a small number of human labels. No iterative prompt search, no logit access, and no multi-round LLM interaction are needed. Yet SAJA matches far more complex systems across four evaluation paradigms: 86{\%} F1 on MT-Bench pairwise preference (vs. 78{\%} uncalibrated), competitive performance on five classification benchmarks with a single call, and +5.71{\%} F1 over prompt-optimized baselines on proprietary data. Ablations confirm that multi-dimensional rubric features outperform one-dimensional calibration (SummEval $\rho$ improves from 0.60 to 0.74) and that coarse rubric outputs recover the same human alignment as full logit distributions ($\rho = 0.36$ vs. 0.37), establishing that logit access is unnecessary for calibrated judge alignment. Moreover, SAJA is model-agnostic: a 9B open-source model with SAJA ($\rho{=}0.70$) surpasses raw GPT-4.1 ($\rho{=}0.60$). Its single-call design yields up to 4.8$\times$ cost savings over per-question approaches."
}