Neha Nayak Kennard


2023

pdf
Causal Matching with Text Embeddings: A Case Study in Estimating the Causal Effects of Peer Review Policies
Raymond Zhang | Neha Nayak Kennard | Daniel Smith | Daniel McFarland | Andrew McCallum | Katherine Keith
Findings of the Association for Computational Linguistics: ACL 2023

A promising approach to estimate the causal effects of peer review policies is to analyze data from publication venues that shift policies from single-blind to double-blind from one year to the next. However, in these settings the content of the manuscript is a confounding variable—each year has a different distribution of scientific content which may naturally affect the distribution of reviewer scores. To address this textual confounding, we extend variable ratio nearest neighbor matching to incorporate text embeddings. We compare this matching method to a widely-used causal method of stratified propensity score matching and a baseline of randomly selected matches. For our case study of the ICLR conference shifting from single- to double-blind review from 2017 to 2018, we find human judges prefer manuscript matches from our method in 70% of cases. While the unadjusted estimate of the average causal effect of reviewers’ scores is -0.25, our method shifts the estimate to -0.17, a slightly smaller difference between the outcomes of single- and double-blind policies. We hope this case study enables exploration of additional text-based causal estimation methods and domains in the future.

2022

pdf
DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Neha Nayak Kennard | Tim O’Gorman | Rajarshi Das | Akshay Sharma | Chhandak Bagchi | Matthew Clinton | Pranay Kumar Yelugam | Hamed Zamani | Andrew McCallum
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined. We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors’ stance towards review arguments. Further, we annotate every review and rebuttal sentence. We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.

2016

pdf
Evaluating Word Embeddings Using a Representative Suite of Practical Tasks
Neha Nayak Kennard | Gabor Angeli | Christopher D. Manning
Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP

pdf
Combining Natural Logic and Shallow Reasoning for Question Answering
Gabor Angeli | Neha Nayak Kennard | Christopher D. Manning
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)