Zhihan Cao
2026
JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge
Zhihan Cao | Fumihito Nishino | Hiroaki Yamada | Ha Thanh Nguyen | Yusuke Miyao | Ken Satoh
Proceedings of the Fifteenth Language Resources and Evaluation Conference
Zhihan Cao | Fumihito Nishino | Hiroaki Yamada | Ha Thanh Nguyen | Yusuke Miyao | Ken Satoh
Proceedings of the Fifteenth Language Resources and Evaluation Conference
We introduce JBE-QA, a Japanese Bar Exam Question–Answering dataset to evaluate large language models’ legal knowledge. Derived from the multiple-choice (tantō-shiki) section of the Japanese bar exam (2015–2024), JBE-QA provides the first comprehensive benchmark for Japanese legal-domain evaluation of LLMs. It covers the Civil Code, the Penal Code, and the Constitution, extending beyond the Civil Code focus of prior Japanese resources. Each question is decomposed into independent true/false judgments with structured contextual fields. The dataset contains 3,464 items with balanced labels. We evaluate 26 LLMs, including proprietary, open-weight, Japanese-specialised, and reasoning models. Our results show that proprietary models with reasoning enabled perform best, and the Constitution questions are generally easier than the Civil Code or the Penal Code questions.
2025
On the Distinctive Co-occurrence Characteristics of Antonymy
Zhihan Cao | Hiroaki Yamada | Takenobu Tokunaga
Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025)
Zhihan Cao | Hiroaki Yamada | Takenobu Tokunaga
Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025)
Antonymy has long received particular attention in lexical semantics.Previous studies have shown that antonym pairs frequently co-occur in text, across genres and parts of speech, more often than would be expected by chance. However, whether this co-occurrence pattern is distinctive of antonymy remains unclear, due to a lack of comparison with other semantic relations. This work fills the gap by comparing antonymy with three other relations across parts of speech using robust co-occurrence metrics. We find that antonymy is distinctive in three respects: antonym pairs co-occur with high strength, in a preferred linear order, and within short spans. All results are available online.