@inproceedings{purushothama-etal-2025-ready,
    title = "Not ready for the bench: {LLM} legal interpretation is unstable and uncalibrated to human judgments",
    author = "Purushothama, Abhishek  and
      Min, Junghyun  and
      Waldon, Brandon  and
      Schneider, Nathan",
    editor = "Aletras, Nikolaos  and
      Chalkidis, Ilias  and
      Barrett, Leslie  and
      Goanț{\u{a}}, C{\u{a}}t{\u{a}}lina  and
      Preoțiuc-Pietro, Daniel  and
      Spanakis, Gerasimos",
    booktitle = "Proceedings of the Natural Legal Language Processing Workshop 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.nllp-1.22/",
    pages = "317--317",
    ISBN = "979-8-89176-338-8",
    abstract = "Legal interpretation frequently involves assessing how a legal text, as understood by an `ordinary' speaker of the language, applies to the set of facts characterizing a legal dispute. Recent scholarship has proposed that legal practitioners add large language models (LLMs) to their interpretive toolkit. This work offers an empirical argument against LLM-assisted interpretation as recently practiced by legal scholars and federal judges. Our investigation in English shows that models do not provide stable interpretive judgments and are susceptible to subtle variations in the prompt. While instruction tuning slightly improves model calibration to human judgments, even the best-calibrated LLMs remain weak predictors of human native speakers' judgments."
}