Pretrained Language Models v. Court Ruling Predictions: A Case Study on a Small Dataset of French Court of Appeal Rulings
Olivia Vaudaux, Caroline Bazzoli, Maximin Coavoux, Géraldine Vial, Étienne Vergès
Abstract
NLP systems are increasingly used in the law domain, either by legal institutions or by the industry. As a result there is a pressing need to characterize their strengths and weaknesses and understand their inner workings. This article presents a case study on the task of judicial decision prediction, on a small dataset from French Courts of Appeal. Specifically, our dataset of around 1000 decisions is about the habitual place of residency of children from divorced parents. The task consists in predicting, from the facts and reasons of the documents, whether the court rules that children should live with their mother or their father. Instead of feeding the whole document to a classifier, we carefully construct the dataset to make sure that the input to the classifier does not contain any ‘spoilers’ (it is often the case in court rulings that information all along the document mentions the final decision). Our results are mostly negative: even classifiers based on French pretrained language models (Flaubert, JuriBERT) do not classify the decisions with a reasonable accuracy. However, they can extract the decision when it is part of the input. With regards to these results, we argue that there is a strong caveat when constructing legal NLP datasets automatically.- Anthology ID:
- 2023.nllp-1.5
- Volume:
- Proceedings of the Natural Legal Language Processing Workshop 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Daniel Preoțiuc-Pietro, Catalina Goanta, Ilias Chalkidis, Leslie Barrett, Gerasimos (Jerry) Spanakis, Nikolaos Aletras
- Venues:
- NLLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 38–43
- Language:
- URL:
- https://aclanthology.org/2023.nllp-1.5
- DOI:
- 10.18653/v1/2023.nllp-1.5
- Cite (ACL):
- Olivia Vaudaux, Caroline Bazzoli, Maximin Coavoux, Géraldine Vial, and Étienne Vergès. 2023. Pretrained Language Models v. Court Ruling Predictions: A Case Study on a Small Dataset of French Court of Appeal Rulings. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 38–43, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Pretrained Language Models v. Court Ruling Predictions: A Case Study on a Small Dataset of French Court of Appeal Rulings (Vaudaux et al., NLLP-WS 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2023.nllp-1.5.pdf