Boris Katz


2022

pdf
The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank
Adam Yaari | Jan DeWitt | Henry Hu | Bennett Stankovits | Sue Felshin | Yevgeni Berzak | Helena Aparicio | Boris Katz | Ignacio Cases | Andrei Barbu
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

Treebanks have traditionally included only text and were derived from written sources such as newspapers or the web. We introduce the Aligned Multimodal Movie Treebank (AMMT), an English language treebank derived from dialog in Hollywood movies which includes transcriptions of the audio-visual streams with word-level alignment, as well as part of speech tags and dependency parses in the Universal Dependencies formalism. AMMT consists of 31,264 sentences and 218,090 words, that will amount to the 3rd largest UD English treebank and the only multimodal treebank in UD. To help with the web-based annotation effort, we also introduce the Efficient Audio Alignment Annotator (EAAA), a companion tool that enables annotators to significantly speed-up their annotation processes.

2021

pdf
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding
Yen-Ling Kuo | Boris Katz | Andrei Barbu
Findings of the Association for Computational Linguistics: EMNLP 2021

Humans are remarkably flexible when understanding new sentences that include combinations of concepts they have never encountered before. Recent work has shown that while deep networks can mimic some human language abilities when presented with novel sentences, systematic variation uncovers the limitations in the language-understanding abilities of networks. We demonstrate that these limitations can be overcome by addressing the generalization challenges in the gSCAN dataset, which explicitly measures how well an agent is able to interpret novel linguistic commands grounded in vision, e.g., novel pairings of adjectives and nouns. The key principle we employ is compositionality: that the compositional structure of networks should reflect the compositional structure of the problem domain they address, while allowing other parameters to be learned end-to-end. We build a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains. Crucially, our network has the same state-of-the-art performance as prior work while generalizing its knowledge when prior work does not. Our network also provides a level of interpretability that enables users to inspect what each part of networks learns. Robust grounded language understanding without dramatic failures and without corner cases is critical to building safe and fair robots; we demonstrate the significant role that compositionality can play in achieving that goal.

pdf
Measuring Social Biases in Grounded Vision and Language Embeddings
Candace Ross | Boris Katz | Andrei Barbu
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

We generalize the notion of measuring social biases in word embeddings to visually grounded word embeddings. Biases are present in grounded embeddings, and indeed seem to be equally or more significant than for ungrounded embeddings. This is despite the fact that vision and language can suffer from different biases, which one might hope could attenuate the biases in both. Multiple ways exist to generalize metrics measuring bias in word embeddings to this new setting. We introduce the space of generalizations (Grounded-WEAT and Grounded-SEAT) and demonstrate that three generalizations answer different yet important questions about how biases, language, and vision interact. These metrics are used on a new dataset, the first for grounded bias, created by augmenting standard linguistic bias benchmarks with 10,228 images from COCO, Conceptual Captions, and Google Images. Dataset construction is challenging because vision datasets are themselves very biased. The presence of these biases in systems will begin to have real-world consequences as they are deployed, making carefully measuring bias and then mitigating it critical to building a fair society.

2018

pdf
Grounding language acquisition by training semantic parsers using captioned videos
Candace Ross | Andrei Barbu | Yevgeni Berzak | Battushig Myanganbayar | Boris Katz
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

We develop a semantic parser that is trained in a grounded setting using pairs of videos captioned with sentences. This setting is both data-efficient, requiring little annotation, and similar to the experience of children where they observe their environment and listen to speakers. The semantic parser recovers the meaning of English sentences despite not having access to any annotated sentences. It does so despite the ambiguity inherent in vision where a sentence may refer to any combination of objects, object properties, relations or actions taken by any agent in a video. For this task, we collected a new dataset for grounded language acquisition. Learning a grounded semantic parser — turning sentences into logical forms using captioned videos — can significantly expand the range of data that parsers can be trained on, lower the effort of training a semantic parser, and ultimately lead to a better understanding of child language acquisition.

pdf
Assessing Language Proficiency from Eye Movements in Reading
Yevgeni Berzak | Boris Katz | Roger Levy
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)

We present a novel approach for determining learners’ second language proficiency which utilizes behavioral traces of eye movements during reading. Our approach provides stand-alone eyetracking based English proficiency scores which reflect the extent to which the learner’s gaze patterns in reading are similar to those of native English speakers. We show that our scores correlate strongly with standardized English proficiency tests. We also demonstrate that gaze information can be used to accurately predict the outcomes of such tests. Our approach yields the strongest performance when the test taker is presented with a suite of sentences for which we have eyetracking data from other readers. However, it remains effective even using eyetracking with sentences for which eye movement data have not been previously collected. By deriving proficiency as an automatic byproduct of eye movements during ordinary reading, our approach offers a potentially valuable new tool for second language proficiency assessment. More broadly, our results open the door to future methods for inferring reader characteristics from the behavioral traces of reading.

2017

pdf
Predicting Native Language from Gaze
Yevgeni Berzak | Chie Nakamura | Suzanne Flynn | Boris Katz
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

A fundamental question in language learning concerns the role of a speaker’s first language in second language acquisition. We present a novel methodology for studying this question: analysis of eye-movement patterns in second language reading of free-form text. Using this methodology, we demonstrate for the first time that the native language of English learners can be predicted from their gaze fixations when reading English. We provide analysis of classifier uncertainty and learned features, which indicates that differences in English reading are likely to be rooted in linguistic divergences across native languages. The presented framework complements production studies and offers new ground for advancing research on multilingualism.

2016

pdf
Universal Dependencies for Learner English
Yevgeni Berzak | Jessica Kenney | Carolyn Spadine | Jing Xian Wang | Lucia Lam | Keiko Sophie Mori | Sebastian Garza | Boris Katz
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf
Learning to Answer Questions from Wikipedia Infoboxes
Alvaro Morales | Varot Premtoon | Cordelia Avery | Sue Felshin | Boris Katz
Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing

pdf
Anchoring and Agreement in Syntactic Annotations
Yevgeni Berzak | Yan Huang | Andrei Barbu | Anna Korhonen | Boris Katz
Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing

2015

pdf
Do You See What I Mean? Visual Resolution of Linguistic Ambiguities
Yevgeni Berzak | Andrei Barbu | Daniel Harari | Boris Katz | Shimon Ullman
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

pdf
Contrastive Analysis with Predictive Power: Typology Driven Estimation of Grammatical Error Distributions in ESL
Yevgeni Berzak | Roi Reichart | Boris Katz
Proceedings of the Nineteenth Conference on Computational Natural Language Learning

2014

pdf
Reconstructing Native Language Typology from Foreign Language Usage
Yevgeni Berzak | Roi Reichart | Boris Katz
Proceedings of the Eighteenth Conference on Computational Natural Language Learning

2006

pdf
Using Semantic Overlap Scoring in Answering TREC Relationship Questions
Gregory Marton | Boris Katz
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

A first step in answering complex questions, such as those in the “Relationship”' task of the Text REtrieval Conference's Question Answering track (TREC/QA), is finding passages likely to contain pieces of the answer---passage retrieval. We introduce semantic overlap scoring, a new passage retrieval algorithm that facilitates credit assignment for inexact matches between query and candidate answer. Our official submission ranked best among fully automatic systems, at 23% F-measure, while the best system, with manual input, reached 28%. We use our Nuggeteer tool to robustly evaluate each component of our Relationship system post hoc. Ablation studies show that semantic overlap scoring achieves significant performance improvements over a standard passage retrieval baseline.

2005

pdf
Using Syntactic Information to Identify Plagiarism
Özlem Uzuner | Boris Katz | Thade Nahnsen
Proceedings of the Second Workshop on Building Educational Applications Using NLP

pdf
A Comparative Study of Language Models for Book and Author Recognition
Özlem Uzuner | Boris Katz
Second International Joint Conference on Natural Language Processing: Full Papers

pdf
Lexical Chains and Sliding Locality Windows in Content-based Text Similarity Detection
Thade Nahnsen | Özlem Uzuner | Boris Katz
Companion Volume to the Proceedings of Conference including Posters/Demos and tutorial abstracts

2004

pdf
Answering Definition Questions with Multiple Knowledge Sources
Wesley Hildebrandt | Boris Katz | Jimmy Lin
Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004

2003

pdf
Extracting Structural Paraphrases from Aligned Monolingual Corpora
Ali Ibrahim | Boris Katz | Jimmy Lin
Proceedings of the Second International Workshop on Paraphrasing

2002

pdf
Annotating the Semantic Web Using Natural Language
Boris Katz | Jimmy Lin
COLING-02: The 2nd Workshop on NLP and XML (NLPXML-2002)

2001

pdf
Gathering Knowledge for a Question Answering System from Heterogeneous Information Sources
Boris Katz | Jimmy Lin | Sue Felshin
Proceedings of the ACL 2001 Workshop on Human Language Technology and Knowledge Management

2000

pdf
REXTOR: A System for Generating Relations from Natural Language
Boris Katz | Jimmy Lin
ACL-2000 Workshop on Recent Advances in Natural Language Processing and Information Retrieval

1988

pdf
Exploiting Lexical Regularities in Designing Natural Language Systems
Boris Katz | Beth Levin
Coling Budapest 1988 Volume 1: International Conference on Computational Linguistics