Alexandra Vtyurina


2023

pdf
Questions about Contracts: Prompt Templates for Structured Answer Generation
Adam Roegiest | Radha Chitta | Jonathan Donnelly | Maya Lash | Alexandra Vtyurina | Francois Longtin
Proceedings of the Natural Legal Language Processing Workshop 2023

Finding the answers to legal questions about specific clauses in contracts is an important analysis in many legal workflows (e.g., understanding market trends, due diligence, risk mitigation) but more important is being able to do this at scale. In this paper, we present an examination of using large language models to produce (partially) structured answers to legal questions; primarily in the form of multiple choice and multiple select. We first show that traditional semantic matching is unable to perform this task at acceptable accuracy and then show how question specific prompts can achieve reasonable accuracy across a range of generative models. Finally, we show that much of this effectiveness can be maintained when generalized prompt templates are used rather than question specific ones.