James H. Martin

Also published as: James H. Martin, James Martin


2023

pdf
How Good Is the Model in Model-in-the-loop Event Coreference Resolution Annotation?
Shafiuddin Rehan Ahmed | Abhijnan Nath | Michael Regan | Adam Pollins | Nikhil Krishnaswamy | James H. Martin
Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)

Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency. To address this, we propose a model-in-the-loop annotation approach for event coreference resolution, where a machine learning model suggests likely corefering event pairs only. We evaluate the effectiveness of this approach by first simulating the annotation process and then, using a novel annotator-centric Recall-Annotation effort trade-off metric, we compare the results of various underlying models and datasets. We finally present a method for obtaining 97% recall while substantially reducing the workload required by a fully manual annotation process.

pdf
2*n is better than n2: Decomposing Event Coreference Resolution into Two Tractable Problems
Shafiuddin Rehan Ahmed | Abhijnan Nath | James H. Martin | Nikhil Krishnaswamy
Findings of the Association for Computational Linguistics: ACL 2023

Event Coreference Resolution (ECR) is the task of linking mentions of the same event either within or across documents. Most mention pairs are not coreferent, yet many that are coreferent can be identified through simple techniques such as lemma matching of the event triggers or the sentences in which they appear. Existing methods for training coreference systems sample from a largely skewed distribution, making it difficult for the algorithm to learn coreference beyond surface matching. Additionally, these methods are intractable because of the quadratic operations needed. To address these challenges, we break the problem of ECR into two parts: a) a heuristic to efficiently filter out a large number of non-coreferent pairs, and b) a training approach on a balanced set of coreferent and non-coreferent mention pairs. By following this approach, we show that we get comparable results to the state of the art on two popular ECR datasets while significantly reducing compute requirements. We also analyze the mention pairs that are “hard” to accurately classify as coreferent or non-coreferentcode repo: \mathtt{github.com/ahmeshaf/lemma\_ce\_coref}.

pdf
Mapping AMR to UMR: Resources for Adapting Existing Corpora for Cross-Lingual Compatibility
Julia Bonn | Skatje Myers | Jens E. L. Van Gysel | Lukas Denk | Meagan Vigus | Jin Zhao | Andrew Cowell | William Croft | Jan Hajič | James H. Martin | Alexis Palmer | Martha Palmer | James Pustejovsky | Zdenka Urešová | Rosa Vallejos | Nianwen Xue
Proceedings of the 21st International Workshop on Treebanks and Linguistic Theories (TLT, GURT/SyntaxFest 2023)

This paper presents detailed mappings between the structures used in Abstract Meaning Representation (AMR) and those used in Uniform Meaning Representation (UMR). These structures include general semantic roles, rolesets, and concepts that are largely shared between AMR and UMR, but with crucial differences. While UMR annotation of new low-resource languages is ongoing, AMR-annotated corpora already exist for many languages, and these AMR corpora are ripe for conversion to UMR format. Rather than focusing on semantic coverage that is new to UMR (which will likely need to be dealt with manually), this paper serves as a resource (with illustrated mappings) for users looking to understand the fine-grained adjustments that have been made to the representation techniques for semantic categoriespresent in both AMR and UMR.

2022

pdf
The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves
Abhijit Suresh | Jennifer Jacobs | Charis Harty | Margaret Perkoff | James H. Martin | Tamara Sumner
Proceedings of the Thirteenth Language Resources and Evaluation Conference

Transcripts of teaching episodes can be effective tools to understand discourse patterns in classroom instruction. According to most educational experts, sustained classroom discourse is a critical component of equitable, engaging, and rich learning environments for students. This paper describes the TalkMoves dataset, composed of 567 human-annotated K-12 mathematics lesson transcripts (including entire lessons or portions of lessons) derived from video recordings. The set of transcripts primarily includes in-person lessons with whole-class discussions and/or small group work, as well as some online lessons. All of the transcripts are human-transcribed, segmented by the speaker (teacher or student), and annotated at the sentence level for ten discursive moves based on accountable talk theory. In addition, the transcripts include utterance-level information in the form of dialogue act labels based on the Switchboard Dialog Act Corpus. The dataset can be used by educators, policymakers, and researchers to understand the nature of teacher and student discourse in K-12 math classrooms. Portions of this dataset have been used to develop the TalkMoves application, which provides teachers with automated, immediate, and actionable feedback about their mathematics instruction.

pdf
Fine-tuning Transformers with Additional Context to Classify Discursive Moves in Mathematics Classrooms
Abhijit Suresh | Jennifer Jacobs | Margaret Perkoff | James H. Martin | Tamara Sumner
Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

“Talk moves” are specific discursive strategies used by teachers and students to facilitate conversations in which students share their thinking, and actively consider the ideas of others, and engage in rich discussions. Experts in instructional practices often rely on cues to identify and document these strategies, for example by annotating classroom transcripts. Prior efforts to develop automated systems to classify teacher talk moves using transformers achieved a performance of 76.32% F1. In this paper, we investigate the feasibility of using enriched contextual cues to improve model performance. We applied state-of-the-art deep learning approaches for Natural Language Processing (NLP), including Robustly optimized bidirectional encoder representations from transformers (Roberta) with a special input representation that supports previous and subsequent utterances as context for talk moves classification. We worked with the publically available TalkMoves dataset, which contains utterances sourced from real-world classroom sessions (human- transcribed and annotated). Through a series of experimentations, we found that a combination of previous and subsequent utterances improved the transformers’ ability to differentiate talk moves (by 2.6% F1). These results constitute a new state of the art over previously published results and provide actionable insights to those in the broader NLP community who are working to develop similar transformer-based classification models.

2020

pdf
Defining and Learning Refined Temporal Relations in the Clinical Narrative
Kristin Wright-Bettner | Chen Lin | Timothy Miller | Steven Bethard | Dmitriy Dligach | Martha Palmer | James H. Martin | Guergana Savova
Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis

We present refinements over existing temporal relation annotations in the Electronic Medical Record clinical narrative. We refined the THYME corpus annotations to more faithfully represent nuanced temporality and nuanced temporal-coreferential relations. The main contributions are in re-defining CONTAINS and OVERLAP relations into CONTAINS, CONTAINS-SUBEVENT, OVERLAP and NOTED-ON. We demonstrate that these refinements lead to substantial gains in learnability for state-of-the-art transformer models as compared to previously reported results on the original THYME corpus. We thus establish a baseline for the automatic extraction of these refined temporal relations. Although our study is done on clinical narrative, we believe it addresses far-reaching challenges that are corpus- and domain- agnostic.

2017

pdf
Abstract Meaning Representation Parsing using LSTM Recurrent Neural Networks
William Foland | James H. Martin
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

We present a system which parses sentences into Abstract Meaning Representations, improving state-of-the-art results for this task by more than 5%. AMR graphs represent semantic content using linguistic properties such as semantic roles, coreference, negation, and more. The AMR parser does not rely on a syntactic pre-parse, or heavily engineered features, and uses five recurrent neural networks as the key architectural components for inferring AMR graphs.

2016

pdf
A Tangled Web: The Faint Signals of Deception in Text - Boulder Lies and Truth Corpus (BLT-C)
Franco Salvetti | John B. Lowe | James H. Martin
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)

We present an approach to creating corpora for use in detecting deception in text, including a discussion of the challenges peculiar to this task. Our approach is based on soliciting several types of reviews from writers and was implemented using Amazon Mechanical Turk. We describe the multi-dimensional corpus of reviews built using this approach, available free of charge from LDC as the Boulder Lies and Truth Corpus (BLT-C). Challenges for both corpus creation and the deception detection include the fact that human performance on the task is typically at chance, that the signal is faint, that paid writers such as turkers are sometimes deceptive, and that deception is a complex human behavior; manifestations of deception depend on details of domain, intrinsic properties of the deceiver (such as education, linguistic competence, and the nature of the intention), and specifics of the deceptive act (e.g., lying vs. fabricating.) To overcome the inherent lack of ground truth, we have developed a set of semi-automatic techniques to ensure corpus validity. We present some preliminary results on the task of deception detection which suggest that the BLT-C is an improvement in the quality of resources available for this task.

pdf
CU-NLP at SemEval-2016 Task 8: AMR Parsing using LSTM-based Recurrent Neural Networks
William Foland | James H. Martin
Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)

2015

pdf
SGRank: Combining Statistical and Graphical Methods to Improve the State of the Art in Unsupervised Keyphrase Extraction
Soheil Danesh | Tamara Sumner | James H. Martin
Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics

pdf
Dependency-Based Semantic Role Labeling using Convolutional Neural Networks
William Foland | James Martin
Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics

2012

pdf
Foundations of a Multilayer Annotation Framework for Twitter Communications During Crisis Events
William J. Corvey | Sudha Verma | Sarah Vieweg | Martha Palmer | James H. Martin
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)

In times of mass emergency, vast amounts of data are generated via computer-mediated communication (CMC) that are difficult to manually collect and organize into a coherent picture. Yet valuable information is broadcast, and can provide useful insight into time- and safety-critical situations if captured and analyzed efficiently and effectively. We describe a natural language processing component of the EPIC (Empowering the Public with Information in Crisis) Project infrastructure, designed to extract linguistic and behavioral information from tweet text to aid in the task of information integration. The system incorporates linguistic annotation, in the form of Named Entity Tagging, as well as behavioral annotations to capture tweets contributing to situational awareness and analyze the information type of the tweet content. We show classification results and describe future integration of these classifiers in the larger EPIC infrastructure.

pdf bib
Identifying science concepts and student misconceptions in an interactive essay writing tutor
Steven Bethard | Ifeyinwa Okoye | Md. Arafat Sultan | Haojie Hang | James H. Martin | Tamara Sumner
Proceedings of the Seventh Workshop on Building Educational Applications Using NLP

2009

pdf bib
Topic Model Analysis of Metaphor Frequency for Psycholinguistic Stimuli
Steven Bethard | Vicky Tzuyin Lai | James H. Martin
Proceedings of the Workshop on Computational Approaches to Linguistic Creativity

2008

pdf
Pedagogically Useful Extractive Summaries for Science Education
Sebastian de la Chica | Faisal Ahmad | James H. Martin | Tamara Sumner
Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)

pdf
Building a Corpus of Temporal-Causal Structure
Steven Bethard | William Corvey | Sara Klingenstein | James H. Martin
Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)

While recent corpus annotation efforts cover a wide variety of semantic structures, work on temporal and causal relations is still in its early stages. Annotation efforts have typically considered either temporal relations or causal relations, but not both, and no corpora currently exist that allow the relation between temporals and causals to be examined empirically. We have annotated a corpus of 1000 event pairs for both temporal and causal relations, focusing on a relatively frequent construction in which the events are conjoined by the word “and”. Temporal relations were annotated using an extension of the BEFORE and AFTER scheme used in the TempEval competition, and causal relations were annotated using a scheme based on connective phrases like “and as a result”. The annotators achieved 81.2% agreement on temporal relations and 77.8% agreement on causal relations. Analysis of the resulting corpus revealed some interesting findings, for example, that over 30% of CAUSAL relations do not have an underlying BEFORE relation. The corpus was also explored using machine learning methods, and while model performance exceeded all baselines, the results suggested that simple grammatical cues may be insufficient for identifying the more difficult temporal and causal relations.

pdf
Annotating Students’ Understanding of Science Concepts
Rodney D. Nielsen | Wayne Ward | James Martin | Martha Palmer
Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)

This paper summarizes the annotation of fine-grained entailment relationships in the context of student answers to science assessment questions. We annotated a corpus of 15,357 answer pairs with 145,911 fine-grained entailment relationships. We provide the rationale for such fine-grained analysis and discuss its perceived benefits to an Intelligent Tutoring System. The corpus also has potential applications in other areas, such as question answering and multi-document summarization. Annotators achieved 86.2% inter-annotator agreement (Kappa=0.728, corresponding to substantial agreement) annotating the fine-grained facets of reference answers with regard to understanding expressed in student answers and labeling from one of five possible detailed relationship categories. The corpus described in this paper, which is the only one providing such detailed entailment annotations, is available as a public resource for the research community. The corpus is expected to enable application development, not only for intelligent tutoring systems, but also for general textual entailment applications, that is currently not practical.

pdf
Extractive Summaries for Educational Science Content
Sebastian de la Chica | Faisal Ahmad | James H. Martin | Tamara Sumner
Proceedings of ACL-08: HLT, Short Papers

pdf
Learning Semantic Links from a Corpus of Parallel Temporal and Causal Relations
Steven Bethard | James H. Martin
Proceedings of ACL-08: HLT, Short Papers

pdf
Extracting a Representation from Text for Semantic Analysis
Rodney D. Nielsen | Wayne Ward | James H. Martin | Martha Palmer
Proceedings of ACL-08: HLT, Short Papers

pdf
Towards Robust Semantic Role Labeling
Sameer S. Pradhan | Wayne Ward | James H. Martin
Computational Linguistics, Volume 34, Number 2, June 2008 - Special Issue on Semantic Role Labeling

pdf bib
Classification Errors in a Domain-Independent Assessment System
Rodney D. Nielsen | Wayne Ward | James H. Martin
Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications

2007

pdf
CU-COMSEM: Exploring Rich Features for Unsupervised Web Personal Name Disambiguation
Ying Chen | James H. Martin
Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)

pdf
CU-TMP: Temporal Relation Classification Using Syntactic and Semantic Features
Steven Bethard | James H. Martin
Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)

pdf
Towards Robust Semantic Role Labeling
Sameer Pradhan | Wayne Ward | James Martin
Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference

pdf
Towards Robust Unsupervised Personal Name Disambiguation
Ying Chen | James Martin
Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)

2006

pdf
Identification of Event Mentions and their Semantic Class
Steven Bethard | James H. Martin
Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing

2005

pdf
Semantic Role Chunking Combining Complementary Syntactic Views
Sameer Pradhan | Kadri Hacioglu | Wayne Ward | James H. Martin | Daniel Jurafsky
Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005)

pdf
Semantic Role Labeling Using Different Syntactic Views
Sameer Pradhan | Wayne Ward | Kadri Hacioglu | James Martin | Daniel Jurafsky
Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)

2004

pdf
Shallow Semantic Parsing using Support Vector Machines
Sameer S. Pradhan | Wayne H. Ward | Kadri Hacioglu | James H. Martin | Dan Jurafsky
Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004

pdf
Parsing Arguments of Nominalizations in English and Chinese
Sameer Pradhan | Honglin Sun | Wayne Ward | James H. Martin | Daniel Jurafsky
Proceedings of HLT-NAACL 2004: Short Papers

pdf
Semantic Role Labeling by Tagging Syntactic Chunks
Kadri Hacioglu | Sameer Pradhan | Wayne Ward | James H. Martin | Daniel Jurafsky
Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004

1997

pdf
Contextual Spelling Correction Using Latent Semantic Analysis
Michael P. Jones | James H. Martin
Fifth Conference on Applied Natural Language Processing

1995

pdf bib
Expressing Rhetorical Relations in Instructional Text: a case study of the purpose relation
Keith Vander Linden | James Martin
Computational Linguistics, Volume 21, Number 1, March 1995

1992

pdf
Knowledge Representation and Metaphor
James Martin
Computational Linguistics, Volume 18, Number 1, March 1992

1991

pdf
Conventional Metaphor and the Lexicon
James H. Martin
Lexical Semantics and Knowledge Representation

1988

pdf
The Berkeley Unix Consultant Project
Robert Wilensky | David N. Chin | Marc Luria | James Martin | James Mayfield | Dekai Wu
Computational Linguistics, Volume 14, Number 4, December 1988, LFP: A Logic for Linguistic Descriptions and an Analysis of its Complexity

pdf
Representing Regularities in the Metaphoric Lexicon
James H. Martin
Coling Budapest 1988 Volume 1: International Conference on Computational Linguistics