@inproceedings{shi-etal-2026-aj,
    title = "{AJ}-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation",
    author = "Shi, Wentao  and
      Wang, Yu  and
      Zhao, Yuyang  and
      Chen, Yuxin  and
      Feng, Fuli  and
      Hao, Xueyuan  and
      Su, Xi  and
      GU, Qi  and
      Su, Hui  and
      Cai, Xunliang  and
      He, Xiangnan",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1269/",
    pages = "25371--25413",
    ISBN = "979-8-89176-395-1",
    abstract = "As reinforcement learning continues to scale the training of large language model{--}based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored.We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains{---}search, data systems, and graphical user interfaces{---}comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/."
}