Morgan A. Gray
2026
Asking the Right Questions: Can expert-prompted LLMs reformulate legal queries from non-experts?
Katherine Atwell | Morgan A. Gray | Jaromir Savelka | Len Rial | Sera Linardi | Malihe Alikhani
Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)
Katherine Atwell | Morgan A. Gray | Jaromir Savelka | Len Rial | Sera Linardi | Malihe Alikhani
Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)
Large language models are widely used by everyday users, and can be asked to perform tasks that require specialized expertise, such as interpreting contractual terms and conditions, filing personal taxes, or diagnosing medical symptoms. Although these tools should not be used in place of professional advice, they can be useful starting points for users seeking professional help, improving users’ access and interactions with professionals. In this vein, this paper introduces a legal question reformulation task to assist non-experts in their interactions with lawyers. This has the potential to streamline discussions between lawyers and clients, who may not know the correct legal language to communicate their needs. Using a novel evaluation framework informed by legal expertise, we investigate the quality of model-generated legal question reformulations on in-the-wild data from non-experts seeking legal advice. Our findings indicate that LLMs have significant potential in legal reasoning, but some unexpected safety concerns may emerge. Further, adding linguisticallyaligned in-domain text samples can improve performance for smaller models, even when the samples are not aligned factually with the given question.