@inproceedings{zhang-etal-2024-ljpcheck,
    title = "{LJPC}heck: Functional Tests for Legal Judgment Prediction",
    author = "Zhang, Yuan  and
      Huang, Wanhong  and
      Feng, Yi  and
      Li, Chuanyi  and
      Fei, Zhiwei  and
      Ge, Jidong  and
      Luo, Bin  and
      Ng, Vincent",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-acl.350/",
    doi = "10.18653/v1/2024.findings-acl.350",
    pages = "5878--5894",
    abstract = "Legal Judgment Prediction (LJP) refers to the task of automatically predicting judgment results (e.g., charges, law articles and term of penalty) given the fact description of cases. While SOTA models have achieved high accuracy and F1 scores on public datasets, existing datasets fail to evaluate specific aspects of these models (e.g., legal fairness, which significantly impact their applications in real scenarios). Inspired by functional testing in software engineering, we introduce LJPCHECK, a suite of functional tests for LJP models, to comprehend LJP models' behaviors and offer diagnostic insights. We illustrate the utility of LJPCHECK on five SOTA LJP models. Extensive experiments reveal vulnerabilities in these models, prompting an in-depth discussion into the underlying reasons of their shortcomings."
}