Frank Schilder


2023

pdf
Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks
Fangyi Yu | Lee Quartey | Frank Schilder
Findings of the Association for Computational Linguistics: ACL 2023

The use of large language models (LLMs) for zero- or few-shot prompting in natural language processing has given rise to a new research area known as prompt engineering. Recent studies have demonstrated that Chain-of-Thought (CoT) prompts can lead to significant improvements in tasks such as arithmetic and common-sense reasoning. This paper explores the use of such approaches in legal reasoning tasks by conducting experiments on the COLIEE entailment task, which is based on the Japanese Bar exam. We evaluate zero-shot/few-shot and fine-tuning approaches with and without explanations, as well as various prompting strategies. Our results indicate that while CoT prompting and fine-tuning with explanations can improve performance, the best results are achieved with prompts derived from specific legal reasoning techniques, such as IRAC (Issue, Rule, Application, Conclusion). In addition, we observe that few-shot learning where the demonstrations are derived from clustering past training data consistently yields high performance on the COLIEE entailment task for both the years of the data that we tested. Through our experiments, we improve the previous best result on the 2021 COLIEE task from 0.7037 to 0.8025 and surpass the best system from 2022 with an accuracy of 0.789.

2019

pdf
Litigation Analytics: Case Outcomes Extracted from US Federal Court Dockets
Thomas Vacek | Ronald Teo | Dezhao Song | Timothy Nugent | Conner Cowling | Frank Schilder
Proceedings of the Natural Legal Language Processing Workshop 2019

Dockets contain a wealth of information for planning a litigation strategy, but the information is locked up in semi-structured text. Manually deriving the outcomes for each party (e.g., settlement, verdict) would be very labor intensive. Having such information available for every past court case, however, would be very useful for developing a strategy because it potentially reveals tendencies and trends of judges and courts and the opposing counsel. We used Natural Language Processing (NLP) techniques and deep learning methods allowing us to scale the automatic analysis of millions of US federal court dockets. The automatically extracted information is fed into a Litigation Analytics tool that is used by lawyers to plan how they approach concrete litigations.

pdf
Litigation Analytics: Extracting and querying motions and orders from US federal courts
Thomas Vacek | Dezhao Song | Hugo Molina-Salgado | Ronald Teo | Conner Cowling | Frank Schilder
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)

Legal litigation planning can benefit from statistics collected from past decisions made by judges. Information on the typical duration for a submitted motion, for example, can give valuable clues for developing a successful strategy. Such information is encoded in semi-structured documents called dockets. In order to extract and aggregate this information, we deployed various information extraction and machine learning techniques. The aggregated data can be queried in real time within the Westlaw Edge search engine. In addition to a keyword search for judges, lawyers, law firms, parties and courts, we also implemented a question answering interface that offers targeted questions in order to get to the respective answers quicker.

2018

pdf
The E2E NLG Challenge: A Tale of Two Systems
Charese Smiley | Elnaz Davoodi | Dezhao Song | Frank Schilder
Proceedings of the 11th International Conference on Natural Language Generation

This paper presents the two systems we entered into the 2017 E2E NLG Challenge: TemplGen, a templated-based system and SeqGen, a neural network-based system. Through the automatic evaluation, SeqGen achieved competitive results compared to the template-based approach and to other participating systems as well. In addition to the automatic evaluation, in this paper we present and discuss the human evaluation results of our two systems.

2017

pdf
Say the Right Thing Right: Ethics Issues in Natural Language Generation Systems
Charese Smiley | Frank Schilder | Vassilis Plachouras | Jochen L. Leidner
Proceedings of the First ACL Workshop on Ethics in Natural Language Processing

We discuss the ethical implications of Natural Language Generation systems. We use one particular system as a case study to identify and classify issues, and we provide an ethics checklist, in the hope that future system designers may benefit from conducting their own ethics reviews based on our checklist.

pdf
Finding the “right” answers for customers
Frank Schilder
Proceedings of the 10th International Conference on Natural Language Generation

This talk will present a few NLG systems developed within Thomson Reuters providing information to professionals such as lawyers, accountants or traders. Based on the experience developing these system, I will discuss the usefulness of automatic metrics, crowd-sourced evaluation, corpora studies and expert reviews. I will conclude with exploring the question of whether developers of NLG systems need to follow ethical guidelines and how those guidelines could be established.

2016

pdf
When to Plummet and When to Soar: Corpus Based Verb Selection for Natural Language Generation
Charese Smiley | Vassilis Plachouras | Frank Schilder | Hiroko Bretz | Jochen Leidner | Dezhao Song
Proceedings of the 9th International Natural Language Generation conference

2015

pdf
Natural Language Question Answering and Analytics for Diverse and Interlinked Datasets
Dezhao Song | Frank Schilder | Charese Smiley | Chris Brew
Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations

2013

pdf
Domain Adaptable Semantic Clustering in Statistical NLG
Blake Howald | Ravikumar Kondadadi | Frank Schilder
Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers

pdf
GenNext: A Consolidated Domain Adaptable NLG System
Frank Schilder | Blake Howald | Ravi Kondadadi
Proceedings of the 14th European Workshop on Natural Language Generation

pdf
A Statistical NLG Framework for Aggregated Planning and Realization
Ravi Kondadadi | Blake Howald | Frank Schilder
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

2010

pdf
Hunting for the Black Swan: Risk Mining from Text
Jochen Leidner | Frank Schilder
Proceedings of the ACL 2010 System Demonstrations

pdf
Book Review: Representation and Management of Narrative Information: Theoretical Principles and Implementation by Gian Piero Zarri
Frank Schilder
Computational Linguistics, Volume 36, Number 1, March 2010

2008

pdf
FastSum: Fast and Accurate Query-based Multi-document Summarization
Frank Schilder | Ravikumar Kondadadi
Proceedings of ACL-08: HLT, Short Papers

2007

pdf
SemEval-2007 Task 15: TempEval Temporal Relation Identification
Marc Verhagen | Robert Gaizauskas | Frank Schilder | Mark Hepple | Graham Katz | James Pustejovsky
Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)

2001

pdf
From Temporal Expressions To Temporal Information: Semantic Tagging Of News Messages
Frank Schilder | Christopher Habel
Proceedings of the ACL 2001 Workshop on Temporal and Spatial Information Processing

1999

pdf
Pointing to Events
Frank Schilder
Ninth Conference of the European Chapter of the Association for Computational Linguistics

pdf
Reference Hashed
Frank Schilder
The Relation of Discourse/Dialogue Structure and Reference

1998

pdf
An Underspecified Segmented Discourse Representation Theory (USDRT)
Frank Schilder
COLING 1998 Volume 2: The 17th International Conference on Computational Linguistics

pdf
Coherence in Spoken Discourse
Heike Tappe | Frank Schilder
COLING 1998 Volume 2: The 17th International Conference on Computational Linguistics

pdf
An Underspecified Segmented Discourse Representation Theory (USDRT)
Frank Schilder
36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2

pdf
Coherence in Spoken Discourse
Heike Tappe | Frank Schilder
36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2

pdf
Temporal Discourse Markers and the Flow of Events
Frank Schilder
Discourse Relations and Discourse Markers

1995

pdf
Aspect and Discourse Structure: Is a Neutral Viewpoint Required?
Frank Schilder
33rd Annual Meeting of the Association for Computational Linguistics