@article{cao-etal-2026-jbe,
    title = "{JBE}-{QA}: {J}apanese Bar Exam {QA} Dataset for Assessing Legal Domain Knowledge",
    author = "Cao, Zhihan  and
      Nishino, Fumihito  and
      Yamada, Hiroaki  and
      Nguyen, Ha Thanh  and
      Miyao, Yusuke  and
      Satoh, Ken",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    journal = "International Conference on Language Resources and Evaluation",
    volume = "main",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.lrec-main.415/",
    pages = "5317--5327",
    abstract = "We introduce JBE-QA, a Japanese Bar Exam Question{--}Answering dataset to evaluate large language models' legal knowledge. Derived from the multiple-choice (tant{\={o}}-shiki) section of the Japanese bar exam (2015{--}2024), JBE-QA provides the first comprehensive benchmark for Japanese legal-domain evaluation of LLMs. It covers the Civil Code, the Penal Code, and the Constitution, extending beyond the Civil Code focus of prior Japanese resources. Each question is decomposed into independent true/false judgments with structured contextual fields. The dataset contains 3,464 items with balanced labels. We evaluate 26 LLMs, including proprietary, open-weight, Japanese-specialised, and reasoning models. Our results show that proprietary models with reasoning enabled perform best, and the Constitution questions are generally easier than the Civil Code or the Penal Code questions."
}