Giovanni Sartor


2025

pdf bib
Towards Reliable Retrieval in RAG Systems for Large Legal Datasets
Markus Reuter | Tobias Lingenberg | Ruta Liepina | Francesca Lagioia | Marco Lippi | Giovanni Sartor | Andrea Passerini | Burcu Sayin
Proceedings of the Natural Legal Language Processing Workshop 2025

Retrieval-Augmented Generation (RAG) is a promising approach to mitigate hallucinations in Large Language Models (LLMs) for legal applications, but its reliability is critically dependent on the accuracy of the retrieval step. This is particularly challenging in the legal domain, where large databases of structurally similar documents often cause retrieval systems to fail. In this paper, we address this challenge by first identifying and quantifying a critical failure mode we term Document-Level Retrieval Mismatch (DRM), where the retriever selects information from entirely incorrect source documents. To mitigate DRM, we investigate a simple and computationally efficient technique which we refer to as Summary-Augmented Chunking (SAC). This method enhances each text chunk with a document-level synthetic summary, thereby injecting crucial global context that would otherwise be lost during a standard chunking process. Our experiments on a diverse set of legal information retrieval tasks show that SAC greatly reduces DRM and, consequently, also improves text-level retrieval precision and recall. Interestingly, we find that a generic summarization strategy outperforms an approach that incorporates legal expert domain knowledge to target specific legal elements. Our work provides evidence that this practical, scalable, and easily integrable technique enhances the reliability of RAG systems when applied to large-scale legal document datasets.

2024

pdf bib
AMELIA - Argument Mining Evaluation on Legal documents in ItAlian: A CALAMITA Challenge
Giulia Grundler | Andrea Galassi | Piera Santin | Alessia Fidelangeli | Federico Galli | Elena Palmieri | Francesca Lagioia | Giovanni Sartor | Paolo Torroni
Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)

This challenge consists of three classification tasks, in the context of argument mining in the legal domain. The tasks are based on a dataset of 225 Italian decisions on Value Added Tax, annotated to identify and categorize argumentative text. The objective of the first task is to classify each argumentative component as premise or conclusion, while the second and third tasks aim at classifying the type of premise: legal vs factual, and its corresponding argumentation scheme. The classes are highly unbalanced, hence evaluation is based on the macro F1 score.

2022

pdf bib
Detecting Arguments in CJEU Decisions on Fiscal State Aid
Giulia Grundler | Piera Santin | Andrea Galassi | Federico Galli | Francesco Godano | Francesca Lagioia | Elena Palmieri | Federico Ruggeri | Giovanni Sartor | Paolo Torroni
Proceedings of the 9th Workshop on Argument Mining

The successful application of argument mining in the legal domain can dramatically impact many disciplines related to law. For this purpose, we present Demosthenes, a novel corpus for argument mining in legal documents, composed of 40 decisions of the Court of Justice of the European Union on matters of fiscal state aid. The annotation specifies three hierarchical levels of information: the argumentative elements, their types, and their argument schemes. In our experimental evaluation, we address 4 different classification tasks, combining advanced language models and traditional classifiers.

pdf bib
Combining WordNet and Word Embeddings in Data Augmentation for Legal Texts
Sezen Perçin | Andrea Galassi | Francesca Lagioia | Federico Ruggeri | Piera Santin | Giovanni Sartor | Paolo Torroni
Proceedings of the Natural Legal Language Processing Workshop 2022

Creating balanced labeled textual corpora for complex tasks, like legal analysis, is a challenging and expensive process that often requires the collaboration of domain experts. To address this problem, we propose a data augmentation method based on the combination of GloVe word embeddings and the WordNet ontology. We present an example of application in the legal domain, specifically on decisions of the Court of Justice of the European Union.Our evaluation with human experts confirms that our method is more robust than the alternatives.

2021

pdf bib
A Corpus for Multilingual Analysis of Online Terms of Service
Kasper Drawzeski | Andrea Galassi | Agnieszka Jablonowska | Francesca Lagioia | Marco Lippi | Hans Wolfgang Micklitz | Giovanni Sartor | Giacomo Tagiuri | Paolo Torroni
Proceedings of the Natural Legal Language Processing Workshop 2021

We present the first annotated corpus for multilingual analysis of potentially unfair clauses in online Terms of Service. The data set comprises a total of 100 contracts, obtained from 25 documents annotated in four different languages: English, German, Italian, and Polish. For each contract, potentially unfair clauses for the consumer are annotated, for nine different unfairness categories. We show how a simple yet efficient annotation projection technique based on sentence embeddings could be used to automatically transfer annotations across languages.