Questions about Contracts: Prompt Templates for Structured Answer Generation
Adam Roegiest, Radha Chitta, Jonathan Donnelly, Maya Lash, Alexandra Vtyurina, Francois Longtin
Abstract
Finding the answers to legal questions about specific clauses in contracts is an important analysis in many legal workflows (e.g., understanding market trends, due diligence, risk mitigation) but more important is being able to do this at scale. In this paper, we present an examination of using large language models to produce (partially) structured answers to legal questions; primarily in the form of multiple choice and multiple select. We first show that traditional semantic matching is unable to perform this task at acceptable accuracy and then show how question specific prompts can achieve reasonable accuracy across a range of generative models. Finally, we show that much of this effectiveness can be maintained when generalized prompt templates are used rather than question specific ones.- Anthology ID:
- 2023.nllp-1.8
- Volume:
- Proceedings of the Natural Legal Language Processing Workshop 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Daniel Preoțiuc-Pietro, Catalina Goanta, Ilias Chalkidis, Leslie Barrett, Gerasimos Spanakis, Nikolaos Aletras
- Venues:
- NLLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 62–72
- Language:
- URL:
- https://aclanthology.org/2023.nllp-1.8
- DOI:
- 10.18653/v1/2023.nllp-1.8
- Cite (ACL):
- Adam Roegiest, Radha Chitta, Jonathan Donnelly, Maya Lash, Alexandra Vtyurina, and Francois Longtin. 2023. Questions about Contracts: Prompt Templates for Structured Answer Generation. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 62–72, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Questions about Contracts: Prompt Templates for Structured Answer Generation (Roegiest et al., NLLP-WS 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-05/2023.nllp-1.8.pdf