@inproceedings{hou-etal-2024-gaps,
    title = "Gaps or Hallucinations? Scrutinizing Machine-Generated Legal Analysis for Fine-grained Text Evaluations",
    author = "Hou, Abe  and
      Jurayj, William  and
      Holzenberger, Nils  and
      Blair-Stanek, Andrew  and
      Van Durme, Benjamin",
    editor = "Aletras, Nikolaos  and
      Chalkidis, Ilias  and
      Barrett, Leslie  and
      Goanț{\u{a}}, C{\u{a}}t{\u{a}}lina  and
      Preoțiuc-Pietro, Daniel  and
      Spanakis, Gerasimos",
    booktitle = "Proceedings of the Natural Legal Language Processing Workshop 2024",
    month = nov,
    year = "2024",
    address = "Miami, FL, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.nllp-1.24/",
    doi = "10.18653/v1/2024.nllp-1.24",
    pages = "280--302",
    abstract = "Large Language Models (LLMs) show promise as a writing aid for professionals performing legal analyses. However, LLMs can often hallucinate in this setting, in ways difficult to recognize by non-professionals and existing text evaluation metrics. In this work, we pose the question: when can machine-generated legal analysis be evaluated as acceptable? We introduce the neutral notion of gaps {--} as opposed to hallucinations in a strict erroneous sense {--} to refer to the difference between human-written and machine-generated legal analysis. Gaps do not always equate to invalid generation. Working with legal experts, we consider the CLERC generation task proposed in Hou et al. (2024b), leading to a taxonomy, a fine-grained detector for predicting gap categories, and an annotated dataset for automatic evaluation. Our best detector achieves 67{\%} F1 score and 80{\%} precision on the test set. Employing this detector as an automated metric on legal analysis generated by SOTA LLMs, we find around 80{\%} contain hallucinations of different kinds."
}