David McAllester


2020

pdf
On-The-Fly Information Retrieval Augmentation for Language Models
Hai Wang | David McAllester
Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events

Here we experiment with the use of information retrieval as an augmentation for pre-trained language models. The text corpus used in information retrieval can be viewed as form of episodic memory which grows over time. By augmenting GPT 2.0 with information retrieval we achieve a zero shot 15% relative reduction in perplexity on Gigaword corpus without any re-training. We also validate our IR augmentation on an event co-reference task.

2019

pdf
Evidence Sentence Extraction for Machine Reading Comprehension
Hai Wang | Dian Yu | Kai Sun | Jianshu Chen | Dong Yu | David McAllester | Dan Roth
Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

Remarkable success has been achieved in the last few years on some limited machine reading comprehension (MRC) tasks. However, it is still difficult to interpret the predictions of existing MRC models. In this paper, we focus on extracting evidence sentences that can explain or support the answers of multiple-choice MRC tasks, where the majority of answer options cannot be directly extracted from reference documents. Due to the lack of ground truth evidence sentence labels in most cases, we apply distant supervision to generate imperfect labels and then use them to train an evidence sentence extractor. To denoise the noisy labels, we apply a recently proposed deep probabilistic logic learning framework to incorporate both sentence-level and cross-sentence linguistic indicators for indirect supervision. We feed the extracted evidence sentences into existing MRC models and evaluate the end-to-end performance on three challenging multiple-choice MRC datasets: MultiRC, RACE, and DREAM, achieving comparable or better performance than the same models that take as input the full reference document. To the best of our knowledge, this is the first work extracting evidence sentences for multiple-choice MRC.

2017

pdf
Broad Context Language Modeling as Reading Comprehension
Zewei Chu | Hai Wang | Kevin Gimpel | David McAllester
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers

Progress in text understanding has been driven by large datasets that test particular capabilities, like recent datasets for reading comprehension (Hermann et al., 2015). We focus here on the LAMBADA dataset (Paperno et al., 2016), a word prediction task requiring broader context than the immediate sentence. We view LAMBADA as a reading comprehension problem and apply comprehension models based on neural networks. Though these models are constrained to choose a word from the context, they improve the state of the art on LAMBADA from 7.3% to 49%. We analyze 100 instances, finding that neural network readers perform well in cases that involve selecting a name from the context based on dialogue or discourse cues but struggle when coreference resolution or external knowledge is needed.

pdf
Emergent Predication Structure in Hidden State Vectors of Neural Readers
Hai Wang | Takeshi Onishi | Kevin Gimpel | David McAllester
Proceedings of the 2nd Workshop on Representation Learning for NLP

A significant number of neural architectures for reading comprehension have recently been developed and evaluated on large cloze-style datasets. We present experiments supporting the emergence of “predication structure” in the hidden state vectors of these readers. More specifically, we provide evidence that the hidden state vectors represent atomic formulas 𝛷c where 𝛷 is a semantic property (predicate) and c is a constant symbol entity identifier.

2016

pdf
Who did What: A Large-Scale Person-Centered Cloze Dataset
Takeshi Onishi | Hai Wang | Mohit Bansal | Kevin Gimpel | David McAllester
Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing

2015

pdf
Machine Comprehension with Syntax, Frames, and Semantics
Hai Wang | Mohit Bansal | Kevin Gimpel | David McAllester
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)

pdf
A Synopsis of Morphoid Type Theory
David McAllester
Proceedings of the 14th Meeting on the Mathematics of Language (MoL 2015)

1999

pdf
Relating Probabilistic Grammars and Automata
Steven Abney | David McAllester | Fernando Pereira
Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics