Eric Nyberg

Also published as: Eric H. Nyberg, Eric H. Nyberg III, Eric H. Nyberg, 3rd


GameQA: Gamified Mobile App Platform for Building Multiple-Domain Question-Answering Datasets
Njall Skarphedinsson | Breki Gudmundsson | Steinar Smari | Marta Kristin Larusdottir | Hafsteinn Einarsson | Abuzar Khan | Eric Nyberg | Hrafn Loftsson
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations

The methods used to create many of the well-known Question-Answering (QA) datasets are hard to replicate for low-resource languages. A commonality amongst these methods is hiring annotators to source answers from the internet by querying a single answer source, such as Wikipedia. Applying these methods for low-resource languages can be problematic since there is no single large answer source for these languages. Consequently, this can result in a high ratio of unanswered questions, since the amount of information in any single source is limited. To address this problem, we developed a novel crowd-sourcing platform to gather multiple-domain QA data for low-resource languages. Our platform, which consists of a mobile app and a web API, gamifies the data collection process. We successfully released the app for Icelandic (a low-resource language with about 350,000 native speakers) to build a dataset which rivals large QA datasets for high-resource languages both in terms of size and ratio of answered questions. We have made the platform open source with instructions on how to localize and deploy it to gather data for other low-resource languages.


Open Domain Question Answering with A Unified Knowledge Interface
Kaixin Ma | Hao Cheng | Xiaodong Liu | Eric Nyberg | Jianfeng Gao
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

The retriever-reader framework is popular for open-domain question answering (ODQA) due to its ability to use explicit knowledge.Although prior work has sought to increase the knowledge coverage by incorporating structured knowledge beyond text, accessing heterogeneous knowledge sources through a unified interface remains an open question. While data-to-text generation has the potential to serve as a universal interface for data and text, its feasibility for downstream tasks remains largely unknown. In this work, we bridge this gap and use the data-to-text method as a means for encoding structured knowledge for open-domain question answering. Specifically, we propose a verbalizer-retriever-reader framework for ODQA over data and text where verbalized tables from Wikipedia and graphs from Wikidata are used as augmented knowledge sources. We show that our Unified Data and Text QA, UDT-QA, can effectively benefit from the expanded knowledge index, leading to large gains over text-only baselines. Notably, our approach sets the single-model state-of-the-art on Natural Questions. Furthermore, our analyses indicate that verbalized knowledge is preferred for answer reasoning for both adapted and hot-swap settings.

R3 : Refined Retriever-Reader pipeline for Multidoc2dial
Srijan Bansal | Suraj Tripathi | Sumit Agarwal | Sireesh Gururaja | Aditya Srikanth Veerubhotla | Ritam Dutt | Teruko Mitamura | Eric Nyberg
Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering

In this paper, we present our submission to the DialDoc shared task based on the MultiDoc2Dial dataset. MultiDoc2Dial is a conversational question answering dataset that grounds dialogues in multiple documents. The task involves grounding a user’s query in a document followed by generating an appropriate response. We propose several improvements over the baseline’s retriever-reader architecture to aid in modeling goal-oriented dialogues grounded in multiple documents. Our proposed approach employs sparse representations for passage retrieval, a passage re-ranker, the fusion-in-decoder architecture for generation, and a curriculum learning training paradigm. Our approach shows a 12 point improvement in BLEU score compared to the baseline RAG model.

Table Retrieval May Not Necessitate Table-specific Model Design
Zhiruo Wang | Zhengbao Jiang | Eric Nyberg | Graham Neubig
Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI)

Tables are an important form of structured data for both human and machine readers alike, providing answers to questions that cannot, or cannot easily, be found in texts. Recent work has designed special models and training paradigms for table-related tasks such as table-based question answering and table retrieval. Though effective, they add complexity in both modeling and data acquisition compared to generic text solutions and obscure which elements are truly beneficial. In this work, we focus on the task of table retrieval, and ask: “is table-specific model design necessary for table retrieval, or can a simpler text-based model be effectively used to achieve a similar result?’’ First, we perform an analysis on a table-based portion of the Natural Questions dataset (NQ-table), and find that structure plays a negligible role in more than 70% of the cases. Based on this, we experiment with a general Dense Passage Retriever (DPR) based on text and a specialized Dense Table Retriever (DTR) that uses table-specific model designs. We find that DPR performs well without any table-specific design and training, and even achieves superior results compared to DTR when fine-tuned on properly linearized tables. We then experiment with three modules to explicitly encode table structures, namely auxiliary row/column embeddings, hard attention masks, and soft relation-based attention biases. However, none of these yielded significant improvements, suggesting that table-specific model design may not be necessary for table retrieval.

Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
Kaixin Ma | Hao Cheng | Xiaodong Liu | Eric Nyberg | Jianfeng Gao
Findings of the Association for Computational Linguistics: EMNLP 2022

We propose a novel open-domain question answering (ODQA) framework for answering single/multi-hop questions across heterogeneous knowledge sources.The key novelty of our method is the introduction of the intermediary modules into the current retriever-reader pipeline.Unlike previous methods that solely rely on the retriever for gathering all evidence in isolation,our intermediary performs a chain of reasoning over the retrieved set.Specifically, our method links the retrieved evidence with its related global context into graphs and organizes them into a candidate list of evidence chains.Built upon pretrained language models, our system achieves competitive performance on two ODQA datasets, OTT-QA and NQ, against tables and passages from Wikipedia.In particular, our model substantially outperforms the previous state-of-the-art on OTT-QA with an exact match score of 47.3 (45% relative gain).

Coalescing Global and Local Information for Procedural Text Understanding
Kaixin Ma | Filip Ilievski | Jonathan Francis | Eric Nyberg | Alessandro Oltramari
Proceedings of the 29th International Conference on Computational Linguistics

Procedural text understanding is a challenging language reasoning task that requires models to track entity states across the development of a narrative. We identify three core aspects required for modeling this task, namely the local and global view of the inputs, as well as the global view of outputs. Prior methods have considered a subset of these aspects, which leads to either low precision or low recall. In this paper, we propose a new model Coalescing Global and Local Information (CGLI), which builds entity- and timestep-aware input representations (local input) considering the whole context (global input), and we jointly model the entity states with a structured prediction objective (global output). Thus, CGLI simultaneously optimizes for both precision and recall. Moreover, we extend CGLI with additional output layers and integrate it into a story reasoning framework. Extensive experiments on a popular procedural text understanding dataset show that our model achieves state-of-the-art results, while experiments on a story reasoning benchmark show the positive impact of our model on downstream reasoning.

PRO-CS : An Instance-Based Prompt Composition Technique for Code-Switched Tasks
Srijan Bansal | Suraj Tripathi | Sumit Agarwal | Teruko Mitamura | Eric Nyberg
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

Code-switched (CS) data is ubiquitous in today’s globalized world, but the dearth of annotated datasets in code-switching poses a significant challenge for learning diverse tasks across different language pairs. Parameter-efficient prompt-tuning approaches conditioned on frozen language models have shown promise for transfer learning in limited-resource setups. In this paper, we propose a novel instance-based prompt composition technique, PRO-CS, for CS tasks that combine language and task knowledge. We compare our approach with prompt-tuning and fine-tuning for code-switched tasks on 10 datasets across 4 language pairs. Our model outperforms the prompt-tuning approach by significant margins across all datasets and outperforms or remains at par with fine-tuning by using just 0.18% of total parameters. We also achieve competitive results when compared with the fine-tuned model in the low-resource cross-lingual and cross-task setting, indicating the effectiveness of our approach to incorporate new code-switched tasks.


Building Goal-oriented Document-grounded Dialogue Systems
Xi Chen | Faner Lin | Yeju Zhou | Kaixin Ma | Jonathan Francis | Eric Nyberg | Alessandro Oltramari
Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021)

In this paper, we describe our systems for solving the two Doc2Dial shared task: knowledge identification and response generation. We proposed several pre-processing and post-processing methods, and we experimented with data augmentation by pre-training the models on other relevant datasets. Our best model for knowledge identification outperformed the baseline by 10.5+ f1-score on the test-dev split, and our best model for response generation outperformed the baseline by 11+ Sacrebleu score on the test-dev split.

Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models
Kaixin Ma | Filip Ilievski | Jonathan Francis | Satoru Ozaki | Eric Nyberg | Alessandro Oltramari
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

Commonsense reasoning benchmarks have been largely solved by fine-tuning language models. The downside is that fine-tuning may cause models to overfit to task-specific data and thereby forget their knowledge gained during pre-training. Recent works only propose lightweight model updates as models may already possess useful knowledge from past experience, but a challenge remains in understanding what parts and to what extent models should be refined for a given task. In this paper, we investigate what models learn from commonsense reasoning datasets. We measure the impact of three different adaptation methods on the generalization and accuracy of models. Our experiments with two models show that fine-tuning performs best, by learning both the content and the structure of the task, but suffers from overfitting and limited generalization to novel answers. We observe that alternative adaptation methods like prefix-tuning have comparable accuracy, but generalize better to unseen answers and are more robust to adversarial splits.


Flexible retrieval with NMSLIB and FlexNeuART
Leonid Boytsov | Eric Nyberg
Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)

Our objective is to introduce to the NLP community NMSLIB, describe a new retrieval toolkit FlexNeuART, as well as their integration capabilities. NMSLIB, while being one the fastest k-NN search libraries, is quite generic and supports a variety of distance/similarity functions. Because the library relies on the distance-based structure-agnostic algorithms, it can be further extended by adding new distances. FlexNeuART is a modular, extendible and flexible toolkit for candidate generation in IR and QA applications, which supports mixing of classic and neural ranking signals. FlexNeuART can efficiently retrieve mixed dense and sparse representations (with weights learned from training data), which is achieved by extending NMSLIB. In that, other retrieval systems work with purely sparse representations (e.g., Lucene), purely dense representations (e.g., FAISS and Annoy), or only perform mixing at the re-ranking stage.


Bend but Don’t Break? Multi-Challenge Stress Test for QA Models
Hemant Pugaliya | James Route | Kaixin Ma | Yixuan Geng | Eric Nyberg
Proceedings of the 2nd Workshop on Machine Reading for Question Answering

The field of question answering (QA) has seen rapid growth in new tasks and modeling approaches in recent years. Large scale datasets and focus on challenging linguistic phenomena have driven development in neural models, some of which have achieved parity with human performance in limited cases. However, an examination of state-of-the-art model output reveals that a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. We are interested in more clearly defining the strengths and limitations of leading models across diverse QA challenges, intending to help future researchers with identifying pathways to generalizable performance. We conduct extensive qualitative and quantitative analyses on the results of four models across four datasets and relate common errors to model capabilities. We also illustrate limitations in the datasets we examine and discuss a way forward for achieving generalizable models and datasets that broadly test QA capabilities.

Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering
Kaixin Ma | Jonathan Francis | Quanyang Lu | Eric Nyberg | Alessandro Oltramari
Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing

Non-extractive commonsense QA remains a challenging AI task, as it requires systems to reason about, synthesize, and gather disparate pieces of information, in order to generate responses to queries. Recent approaches on such tasks show increased performance, only when models are either pre-trained with additional information or when domain-specific heuristics are used, without any special consideration regarding the knowledge resource type. In this paper, we perform a survey of recent commonsense QA methods and we provide a systematic analysis of popular knowledge resources and knowledge-integration methods, across benchmarks from multiple commonsense datasets. Our results and analysis show that attention-based injection seems to be a preferable choice for knowledge integration and that the degree of domain overlap, between knowledge bases and datasets, plays a crucial role in determining model success.

Storyboarding of Recipes: Grounded Contextual Generation
Khyathi Chandu | Eric Nyberg | Alan W Black
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Information need of humans is essentially multimodal in nature, enabling maximum exploitation of situated context. We introduce a dataset for sequential procedural (how-to) text generation from images in cooking domain. The dataset consists of 16,441 cooking recipes with 160,479 photos associated with different steps. We setup a baseline motivated by the best performing model in terms of human evaluation for the Visual Story Telling (ViST) task. In addition, we introduce two models to incorporate high level structure learnt by a Finite State Machine (FSM) in neural sequential generation process by: (1) Scaffolding Structure in Decoder (SSiD) (2) Scaffolding Structure in Loss (SSiL). Our best performing model (SSiL) achieves a METEOR score of 0.31, which is an improvement of 0.6 over the baseline model. We also conducted human evaluation of the generated grounded recipes, which reveal that 61% found that our proposed (SSiL) model is better than the baseline model in terms of overall recipes. We also discuss analysis of the output highlighting key important NLP issues for prospective directions.

Pentagon at MEDIQA 2019: Multi-task Learning for Filtering and Re-ranking Answers using Language Inference and Question Entailment
Hemant Pugaliya | Karan Saxena | Shefali Garg | Sheetal Shalini | Prashant Gupta | Eric Nyberg | Teruko Mitamura
Proceedings of the 18th BioNLP Workshop and Shared Task

Parallel deep learning architectures like fine-tuned BERT and MT-DNN, have quickly become the state of the art, bypassing previous deep and shallow learning methods by a large margin. More recently, pre-trained models from large related datasets have been able to perform well on many downstream tasks by just fine-tuning on domain-specific datasets (similar to transfer learning). However, using powerful models on non-trivial tasks, such as ranking and large document classification, still remains a challenge due to input size limitations of parallel architecture and extremely small datasets (insufficient for fine-tuning). In this work, we introduce an end-to-end system, trained in a multi-task setting, to filter and re-rank answers in the medical domain. We use task-specific pre-trained models as deep feature extractors. Our model achieves the highest Spearman’s Rho and Mean Reciprocal Rank of 0.338 and 0.9622 respectively, on the ACL-BioNLP workshop MediQA Question Answering shared-task.

Dr.Quad at MEDIQA 2019: Towards Textual Inference and Question Entailment using contextualized representations
Vinayshekhar Bannihatti Kumar | Ashwin Srinivasan | Aditi Chaudhary | James Route | Teruko Mitamura | Eric Nyberg
Proceedings of the 18th BioNLP Workshop and Shared Task

This paper presents the submissions by TeamDr.Quad to the ACL-BioNLP 2019 shared task on Textual Inference and Question Entailment in the Medical Domain. Our system is based on the prior work Liu et al. (2019) which uses a multi-task objective function for textual entailment. In this work, we explore different strategies for generalizing state-of-the-art language understanding models to the specialized medical domain. Our results on the shared task demonstrate that incorporating domain knowledge through data augmentation is a powerful strategy for addressing challenges posed specialized domains such as medicine.

Sieg at MEDIQA 2019: Multi-task Neural Ensemble for Biomedical Inference and Entailment
Sai Abishek Bhaskar | Rashi Rungta | James Route | Eric Nyberg | Teruko Mitamura
Proceedings of the 18th BioNLP Workshop and Shared Task

This paper presents a multi-task learning approach to natural language inference (NLI) and question entailment (RQE) in the biomedical domain. Recognizing textual inference relations and question similarity can address the issue of answering new consumer health questions by mapping them to Frequently Asked Questions on reputed websites like the NIH. We show that leveraging information from parallel tasks across domains along with medical knowledge integration allows our model to learn better biomedical feature representations. Our final models for the NLI and RQE tasks achieve the 4th and 2nd rank on the shared-task leaderboard respectively.


pdf bib
Towards Inference-Oriented Reading Comprehension: ParallelQA
Soumya Wadhwa | Varsha Embar | Matthias Grabmair | Eric Nyberg
Proceedings of the Workshop on Generalization in the Age of Deep Learning

In this paper, we investigate the tendency of end-to-end neural Machine Reading Comprehension (MRC) models to match shallow patterns rather than perform inference-oriented reasoning on RC benchmarks. We aim to test the ability of these systems to answer questions which focus on referential inference. We propose ParallelQA, a strategy to formulate such questions using parallel passages. We also demonstrate that existing neural models fail to generalize well to this setting.

BioAMA: Towards an End to End BioMedical Question Answering System
Vasu Sharma | Nitish Kulkarni | Srividya Pranavi | Gabriel Bayomi | Eric Nyberg | Teruko Mitamura
Proceedings of the BioNLP 2018 workshop

In this paper, we present a novel Biomedical Question Answering system, BioAMA: “Biomedical Ask Me Anything” on task 5b of the annual BioASQ challenge. In this work, we focus on a wide variety of question types including factoid, list based, summary and yes/no type questions that generate both exact and well-formed ‘ideal’ answers. For summary-type questions, we combine effective IR-based techniques for retrieval and diversification of relevant snippets for a question to create an end-to-end system which achieves a ROUGE-2 score of 0.72 and a ROUGE-SU4 score of 0.71 on ideal answer questions (7% improvement over the previous best model). Additionally, we propose a novel NLI-based framework to answer the yes/no questions. To train the NLI model, we also devise a transfer-learning technique by cross-domain projection of word embeddings. Finally, we present a two-stage approach to address the factoid and list type questions by first generating a candidate set using NER taggers and ranking them using both supervised or unsupervised techniques.

Comparative Analysis of Neural QA models on SQuAD
Soumya Wadhwa | Khyathi Chandu | Eric Nyberg
Proceedings of the Workshop on Machine Reading for Question Answering

The task of Question Answering has gained prominence in the past few decades for testing the ability of machines to understand natural language. Large datasets for Machine Reading have led to the development of neural models that cater to deeper language understanding compared to information retrieval tasks. Different components in these neural architectures are intended to tackle different challenges. As a first step towards achieving generalization across multiple domains, we attempt to understand and compare the peculiarities of existing end-to-end neural models on the Stanford Question Answering Dataset (SQuAD) by performing quantitative as well as qualitative analysis of the results attained by each of them. We observed that prediction errors reflect certain model-specific biases, which we further discuss in this paper.

Code-Mixed Question Answering Challenge: Crowd-sourcing Data and Techniques
Khyathi Chandu | Ekaterina Loginova | Vishal Gupta | Josef van Genabith | Günter Neumann | Manoj Chinnakotla | Eric Nyberg | Alan W. Black
Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching

Code-Mixing (CM) is the phenomenon of alternating between two or more languages which is prevalent in bi- and multi-lingual communities. Most NLP applications today are still designed with the assumption of a single interaction language and are most likely to break given a CM utterance with multiple languages mixed at a morphological, phrase or sentence level. For example, popular commercial search engines do not yet fully understand the intents expressed in CM queries. As a first step towards fostering research which supports CM in NLP applications, we systematically crowd-sourced and curated an evaluation dataset for factoid question answering in three CM languages - Hinglish (Hindi+English), Tenglish (Telugu+English) and Tamlish (Tamil+English) which belong to two language families (Indo-Aryan and Dravidian). We share the details of our data collection process, techniques which were used to avoid inducing lexical bias amongst the crowd workers and other CM specific linguistic properties of the dataset. Our final dataset, which is available freely for research purposes, has 1,694 Hinglish, 2,848 Tamlish and 1,391 Tenglish factoid questions and their answers. We discuss the techniques used by the participants for the first edition of this ongoing challenge.

Extraction Meets Abstraction: Ideal Answer Generation for Biomedical Questions
Yutong Li | Nicholas Gekakis | Qiuze Wu | Boyue Li | Khyathi Chandu | Eric Nyberg
Proceedings of the 6th BioASQ Workshop A challenge on large-scale biomedical semantic indexing and question answering

The growing number of biomedical publications is a challenge for human researchers, who invest considerable effort to search for relevant documents and pinpointed answers. Biomedical Question Answering can automatically generate answers for a user’s topic or question, significantly reducing the effort required to locate the most relevant information in a large document corpus. Extractive summarization techniques, which concatenate the most relevant text units drawn from multiple documents, perform well on automatic evaluation metrics like ROUGE, but score poorly on human readability, due to the presence of redundant text and grammatical errors in the answer. This work moves toward abstractive summarization, which attempts to distill and present the meaning of the original text in a more coherent way. We incorporate a sentence fusion approach, based on Integer Linear Programming, along with three novel approaches for sentence ordering, in an attempt to improve the human readability of ideal answers. Using an open framework for configuration space exploration (BOOM), we tested over 2000 unique system configurations in order to identify the best-performing combinations for the sixth edition of Phase B of the BioASQ challenge.

Ontology-Based Retrieval & Neural Approaches for BioASQ Ideal Answer Generation
Ashwin Naresh Kumar | Harini Kesavamoorthy | Madhura Das | Pramati Kalwad | Khyathi Chandu | Teruko Mitamura | Eric Nyberg
Proceedings of the 6th BioASQ Workshop A challenge on large-scale biomedical semantic indexing and question answering

The ever-increasing magnitude of biomedical information sources makes it difficult and time-consuming for a human researcher to find the most relevant documents and pinpointed answers for a specific question or topic when using only a traditional search engine. Biomedical Question Answering systems automatically identify the most relevant documents and pinpointed answers, given an information need expressed as a natural language question. Generating a non-redundant, human-readable summary that satisfies the information need of a given biomedical question is the focus of the Ideal Answer Generation task, part of the BioASQ challenge. This paper presents a system for ideal answer generation (using ontology-based retrieval and a neural learning-to-rank approach, combined with extractive and abstractive summarization techniques) which achieved the highest ROUGE score of 0.659 on the BioASQ 5b batch 2 test.


Tackling Biomedical Text Summarization: OAQA at BioASQ 5B
Khyathi Chandu | Aakanksha Naik | Aditya Chandrasekar | Zi Yang | Niloy Gupta | Eric Nyberg
BioNLP 2017

In this paper, we describe our participation in phase B of task 5b of the fifth edition of the annual BioASQ challenge, which includes answering factoid, list, yes-no and summary questions from biomedical data. We describe our techniques with an emphasis on ideal answer generation, where the goal is to produce a relevant, precise, non-redundant, query-oriented summary from multiple relevant documents. We make use of extractive summarization techniques to address this task and experiment with different biomedical ontologies and various algorithms including agglomerative clustering, Maximum Marginal Relevance (MMR) and sentence compression. We propose a novel word embedding based tf-idf similarity metric and a soft positional constraint which improve our system performance. We evaluate our techniques on test batch 4 from the fourth edition of the challenge. Our best system achieves a ROUGE-2 score of 0.6534 and ROUGE-SU4 score of 0.6536.

pdf bib
Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models
Harsh Jhamtani | Varun Gangal | Eduard Hovy | Eric Nyberg
Proceedings of the Workshop on Stylistic Variation

Variations in writing styles are commonly used to adapt the content to a specific context, audience, or purpose. However, applying stylistic variations is still by and large a manual process, and there have been little efforts towards automating it. In this paper we explore automated methods to transform text from modern English to Shakespearean English using an end to end trainable neural model with pointers to enable copy action. To tackle limited amount of parallel data, we pre-train embeddings of words by leveraging external dictionaries mapping Shakespearean words to modern English words as well as additional text. Our methods are able to get a BLEU score of 31+, an improvement of ≈ 6 points above the strongest baseline. We publicly release our code to foster further research in this area.

How Would You Say It? Eliciting Lexically Diverse Dialogue for Supervised Semantic Parsing
Abhilasha Ravichander | Thomas Manzini | Matthias Grabmair | Graham Neubig | Jonathan Francis | Eric Nyberg
Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue

Building dialogue interfaces for real-world scenarios often entails training semantic parsers starting from zero examples. How can we build datasets that better capture the variety of ways users might phrase their queries, and what queries are actually realistic? Wang et al. (2015) proposed a method to build semantic parsing datasets by generating canonical utterances using a grammar and having crowdworkers paraphrase them into natural wording. A limitation of this approach is that it induces bias towards using similar language as the canonical utterances. In this work, we present a methodology that elicits meaningful and lexically diverse queries from users for semantic parsing tasks. Starting from a seed lexicon and a generative grammar, we pair logical forms with mixed text-image representations and ask crowdworkers to paraphrase and confirm the plausibility of the queries that they generated. We use this method to build a semantic parsing dataset from scratch for a dialog agent in a smart-home simulation. We find evidence that this dataset, which we have named SmartHome, is demonstrably more lexically diverse and difficult to parse than existing domain-specific semantic parsing datasets.

Structural Embedding of Syntactic Trees for Machine Comprehension
Rui Liu | Junjie Hu | Wei Wei | Zi Yang | Eric Nyberg
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

Deep neural networks for machine comprehension typically utilizes only word or character embeddings without explicitly taking advantage of structured linguistic information such as constituency trees and dependency trees. In this paper, we propose structural embedding of syntactic trees (SEST), an algorithm framework to utilize structured information and encode them into vector representations that can boost the performance of algorithms for the machine comprehension. We evaluate our approach using a state-of-the-art neural attention model on the SQuAD dataset. Experimental results demonstrate that our model can accurately identify the syntactic boundaries of the sentences and extract answers that are syntactically coherent over the baseline methods.

Steering Output Style and Topic in Neural Response Generation
Di Wang | Nebojsa Jojic | Chris Brockett | Eric Nyberg
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

We propose simple and flexible training and decoding methods for influencing output style and topic in neural encoder-decoder based language generation. This capability is desirable in a variety of applications, including conversational systems, where successful agents need to produce language in a specific style and generate responses steered by a human puppeteer or external knowledge. We decompose the neural generation process into empirically easier sub-problems: a faithfulness model and a decoding method based on selective-sampling. We also describe training and sampling algorithms that bias the generation process with a specific language style restriction, or a topic restriction. Human evaluation results show that our proposed methods are able to to restrict style and topic without degrading output quality in conversational tasks.

Charmanteau: Character Embedding Models For Portmanteau Creation
Varun Gangal | Harsh Jhamtani | Graham Neubig | Eduard Hovy | Eric Nyberg
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

Portmanteaus are a word formation phenomenon where two words combine into a new word. We propose character-level neural sequence-to-sequence (S2S) methods for the task of portmanteau generation that are end-to-end-trainable, language independent, and do not explicitly use additional phonetic information. We propose a noisy-channel-style model, which allows for the incorporation of unsupervised word lists, improving performance over a standard source-to-target model. This model is made possible by an exhaustive candidate generation strategy specifically enabled by the features of the portmanteau task. Experiments find our approach superior to a state-of-the-art FST-based baseline with respect to ground truth accuracy and human evaluation.


Learning to Answer Biomedical Questions: OAQA at BioASQ 4B
Zi Yang | Yue Zhou | Eric Nyberg
Proceedings of the Fourth BioASQ workshop

pdf bib
LAPPS/Galaxy: Current State and Next Steps
Nancy Ide | Keith Suderman | Eric Nyberg | James Pustejovsky | Marc Verhagen
Proceedings of the Third International Workshop on Worldwide Language Service Infrastructure and Second Workshop on Open Infrastructures and Analysis Frameworks for Human Language Technologies (WLSI/OIAF4HLT2016)

The US National Science Foundation (NSF) SI2-funded LAPPS/Galaxy project has developed an open-source platform for enabling complex analyses while hiding complexities associated with underlying infrastructure, that can be accessed through a web interface, deployed on any Unix system, or run from the cloud. It provides sophisticated tool integration and history capabilities, a workflow system for building automated multi-step analyses, state-of-the-art evaluation capabilities, and facilities for sharing and publishing analyses. This paper describes the current facilities available in LAPPS/Galaxy and outlines the project’s ongoing activities to enhance the framework.

QGASP: a Framework for Question Generation Based on Different Levels of Linguistic Information
Hugo Patinho Rodrigues | Luísa Coheur | Eric Nyberg
Proceedings of the 9th International Natural Language Generation conference


A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering
Di Wang | Eric Nyberg
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)


Metaphor Detection with Cross-Lingual Model Transfer
Yulia Tsvetkov | Leonid Boytsov | Anatole Gershman | Eric Nyberg | Chris Dyer
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

The Language Application Grid
Nancy Ide | James Pustejovsky | Christopher Cieri | Eric Nyberg | Di Wang | Keith Suderman | Marc Verhagen | Jonathan Wright
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

The Language Application (LAPPS) Grid project is establishing a framework that enables language service discovery, composition, and reuse and promotes sustainability, manageability, usability, and interoperability of natural language Processing (NLP) components. It is based on the service-oriented architecture (SOA), a more recent, web-oriented version of the “pipeline” architecture that has long been used in NLP for sequencing loosely-coupled linguistic analyses. The LAPPS Grid provides access to basic NLP processing tools and resources and enables pipelining such tools to create custom NLP applications, as well as composite services such as question answering and machine translation together with language resources such as mono- and multi-lingual corpora and lexicons that support NLP. The transformative aspect of the LAPPS Grid is that it orchestrates access to and deployment of language resources and processing functions available from servers around the globe and enables users to add their own language resources, services, and even service grids to satisfy their particular needs.


Assessing Benefit from Feature Feedback in Active Learning for Text Classification
Shilpa Arora | Eric Nyberg
Proceedings of the Fifteenth Conference on Computational Natural Language Learning


Sentiment Classification using Automatically Extracted Subgraph Features
Shilpa Arora | Elijah Mayfield | Carolyn Penstein-Rosé | Eric Nyberg
Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text


Estimating Annotation Cost for Active Learning in a Multi-Annotator Environment
Shilpa Arora | Eric Nyberg | Carolyn P. Rosé
Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing

Interactive Annotation Learning with Indirect Feature Voting
Shilpa Arora | Eric Nyberg
Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Student Research Workshop and Doctoral Consortium


pdf bib
Improving Text Retrieval Precision and Answer Accuracy in Question Answering Systems
Matthew Bilotti | Eric Nyberg
Coling 2008: Proceedings of the 2nd workshop on Information Retrieval for Question Answering

Automatic Set Expansion for List Question Answering
Richard C. Wang | Nico Schlaefer | William W. Cohen | Eric Nyberg
Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing


Language-independent Probabilistic Answer Ranking for Question Answering
Jeongwoo Ko | Teruko Mitamura | Eric Nyberg
Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics

A Probabilistic Framework for Answer Selection in Question Answering
Jeongwoo Ko | Luo Si | Eric Nyberg
Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference


Evaluation for Scenario Question Answering Systems
Matthew W. Bilotti | Eric Nyberg
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

Scenario Question Answering is a relatively new direction in Question Answering (QA) research that presents a number of challenges for evaluation. In this paper, we propose a comprehensive evaluation strategy for Scenario QA, including amethodology for building reusable test collections for Scenario QA and metrics for evaluating system performance over such test collections. Using this methodology, we have built a test collection, which we have made available for public download as a service to the research community. It is our hope that widespread availability of quality evaluation materials fuels research in new approaches to the Scenario QA task.

Exploiting Multiple Semantic Resources for Answer Selection
Jeongwoo Ko | Laurie Hiyakumoto | Eric Nyberg
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

This paper describes the utility of semantic resources such as the Web, WordNet and gazetteers in the answer selection process for a question-answering system. In contrast with previous work using individual semantic resources to support answer selection, our work combines multiple resources to boost the confidence scores assigned to correct answers and evaluates different combination strategies based on unweighted sums, weighted linear combinations, and logistic regression. We apply our approach to select answers from candidates produced by three different extraction techniques of varying quality, focusing on TREC questions whose answers represent locations or proper-names. Our experimental results demonstrate that the combination of semantic resources is more effective than individual resources for all three extraction techniques, improving answer selection accuracy by as much as 32.35% for location questions and 72% for proper-name questions. Of the combination strategies tested, logistic regression models produced the best results for both location and proper-name questions.

Analyzing the Effects of Spoken Dialog Systems on Driving Behavior
Jeongwoo Ko | Fumihiko Murase | Teruko Mitamura | Eric Nyberg | Masahiko Tateishi | Ichiro Akahori
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)

This paper presents an evaluation of a spoken dialog system for automotive environments. Our overall goal was to measure the impact of user-system interaction on the user’s driving performance, and to determine whether adding context-awareness to the dialog system might reduce the degree of user distraction during driving. To address this issue, we incorporated context-awareness into a spoken dialog system, and implemented three system features using user context, network context and dialog context. A series of experiments were conducted under three different configurations: driving without a dialog system, driving while using a context-aware dialog system, and driving while using a context-unaware dialog system. We measured the differences between the three configurations by comparing the average car speed, the frequency of speed changes and the angle between the car’s direction and the centerline on the road. These results indicate that context-awareness could reduce the degree of user distraction when using a dialog system during driving.


Pronominal Anaphora Resolution for Unrestricted Text
Anna Kupść | Teruko Mitamura | Benjamin Van Durme | Eric Nyberg
Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)

An Information Repository Model for Advanced Question Answering Systems
Vasco Calais Pedro | Jeongwoo Ko | Eric Nyberg | Teruko Mitamura
Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)

Correction Grammars for Error Handling in a Speech Dialog System
Hirohiko Sagawa | Teruko Mitamura | Eric Nyberg
Proceedings of HLT-NAACL 2004: Short Papers

Resource Analysis for Question Answering
Lucian Vlad Lita | Warren A. Hunt | Eric Nyberg
Proceedings of the ACL Interactive Poster and Demonstration Sessions


Diagnostics for interactive controlled language checking
Teruko Mitamura | Kathryn Baker | Eric Nyberg | David Svoboda
EAMT Workshop: Improving MT through other language technology tools: resources and tools for building MT

JAVELIN: A Flexible, Planner-Based Architecture for Question Answering
Eric Nyberg | Robert Frederking
Companion Volume of the Proceedings of HLT-NAACL 2003 - Demonstrations

Towards light semantic processing for question answering
Benjamin Van Durme | Yifen Huang | Anna Kupść | Eric Nyberg
Proceedings of the HLT-NAACL 2003 Workshop on Text Meaning

Source language diagnostics for MT
Teruko Mitamura | Kathryn Baker | David Svoboda | Eric Nyberg
Proceedings of Machine Translation Summit IX: Papers

This paper presents a source language diagnostic system for controlled translation. Diagnostics were designed and implemented to address the most difficult rewrites for authors, based on an empirical analysis of log files containing over 180,000 sentences. The design and implementation of the diagnostic system are presented, along with experimental results from an empirical evaluation of the completed system. We found that the diagnostic system can correctly identify the problem in 90.2% of the cases. In addition, depending on the type of grammar problem, the diagnostic system may offer a rewritten sentence. We found that 89.4% of the rewritten sentences were correctly rewritten. The results suggest that these methods could be used as the basis for an automatic rewriting system in the future.

An integrated system for source language checking, analysis and term management
Eric Nyberg | Teruko Mitamura | David Svoboda | Jeongwoo Ko | Kathryn Baker | Jeffrey Micher
Proceedings of Machine Translation Summit IX: System Presentations

This paper presents an overview of the tools provided by KANTOO MT system for controlled source language checking, source text analysis, and terminology management. The steps in each process are described, and screen images are provided to illustrate the system architecture and example tool interfaces.

Teaching machine translation in a graduate language technologies program
Teruko Mitamura | Eric Nyberg | Robert Frederking
Workshop on Teaching Translation Technologies and Tools

This paper describes a graduate-level machine translation (MT) course taught at the Language Technologies Institute at Carnegie Mellon University. Most of the students in the course have a background in computer science. We discuss what we teach (the course syllabus), and how we teach it (lectures, homeworks, and projects). The course has evolved steadily over the past several years to incorporate refinements in the set of course topics, how they are taught, and how students “learn by doing”. The course syllabus has also evolved in response to changes in the field of MT and the role that MT plays in various social contexts.


Pronominal anaphora resolution in the KANTOO multilingual machine translation system
Teruko Mitamura | Eric Nyberg | Enrique Torrejon | Dave Svoboda | Annelen Brunner | Kathryn Baker
Proceedings of the 9th Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages: Papers

Design and Evolution of a Language Technologies Curriculum
Robert Frederking | Eric H. Nyberg | Teruko Mitamura | Jaime G. Carbonell
Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics

Deriving semantic knowledge from descriptive texts using an MT system
Eric Nyberg | Teruko Mitamura | Kathryn Baker | David Svoboda | Brian Peterson | Jennifer Williams
Proceedings of the 5th Conference of the Association for Machine Translation in the Americas: Technical Papers

This paper describes the results of a feasibility study which focused on deriving semantic networks from descriptive texts using controlled language. The KANT system [3,6] was used to analyze input paragraphs, producing sentence-level interlingua representations. The interlinguas were merged to construct a paragraph-level representation, which was used to create a semantic network in Conceptual Graph (CG) [1] format. The interlinguas are also translated (using the KANTOO generator) into OWL statements for entry into the Ontology Works electrical power factbase [9]. The system was extended to allow simple querying in natural language.

The KANTOO MT sytem: controlled language checker and lexical maintenance tool
Teriuko Mitamura | Eric Nyberg | Kathy Baker | Peter Cramer | Jeongwoo Ko | David Svoboda | Michael Duggan
Proceedings of the 5th Conference of the Association for Machine Translation in the Americas: System Descriptions

We will present the KANTOO machine translation environment, a set of software servers and tools for multilingual document production. KANTOO includes modules for source language analysis, target language generation, source terminology management, target terminology management, and knowledge source development (see Figure 1).


Pronominal anaphora resolution in KANTOO English-to-Spanish machine translation system
Teruko Mitamura | Eric Nyberg | Enrique Torrejon | David Svoboda | Kathryn Baker
Proceedings of Machine Translation Summit VIII

We describe the automatic resolution of pronominal anaphora using KANT Controlled English (KCE) and the KANTOO English-to-Spanish MT system. Our algorithm is based on a robust, syntax-based approach that applies a set of restrictions and preferences to select the correct antecedent. We report a success rate of 89.6% on a training corpus with 289 anaphors, and 87.5% on held-out data containing 145 anaphors. Resolution of anaphors is important in translation, due to gender mismatches among languages; our approach translates anaphors to Spanish with 97.2% accuracy.

Integrated Information Management: An Interactive, Extensible Architecture for Information Retrieval
Eric Nyberg | Hal Daume
Proceedings of the First International Conference on Human Language Technology Research


Controlled languages
Teruko Mitamura | Eric Nyberg
Proceedings of the Fourth Conference of the Association for Machine Translation in the Americas: Tutorial Descriptions

Challenges in adapting an interlingua for bidirectional English-Italian translation
Violetta Cavalli-Sforza | Krzysztof Czuba | Teruko Mitamura | Eric Nyberg
Proceedings of the Fourth Conference of the Association for Machine Translation in the Americas: Technical Papers

We describe our experience in adapting an existing high- quality, interlingual, unidirectional machine translation system to a new domain and bidirectional translation for a new language pair (English and Italian). We focus on the interlingua design changes which were necessary to achieve high quality output in view of the language mismatches between English and Italian. The representation we propose contains features that are interpreted differently, depending on the translation direction. This decision simplified the process of creating the interlingua for individual sentences, and allows the system to defer mapping of language-specific features (such as tense and aspect), which are realized when the target syntactic feature structure is created. We also describe a set of problems we encountered in translating modal verbs, and discuss the representation of modality in our interlingua.

pdf bib
The KANTOO machine translation environment
Eric Nyberg | Teruko Mitamura
Proceedings of the Fourth Conference of the Association for Machine Translation in the Americas: System Descriptions

In this paper we describe the KANTOO machine translation environment, a set of software services and tools for multilingual document production. KANTOO includes modules for source language analysis, target language generation, source terminology management, target terminology management, and knowledge source development. The KANTOOsystem represents a complete re-design and re-implementation of the KANT machine translation system.


Multiple strategies for automatic disambiguation in technical translation
Teruko Mitamura | Eric Nyberg | Enrique Torrejon | Robert Igo
Proceedings of the 8th Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages


An English-to-Turkish interlingual MT system
Dilek Zeynap Hakkani | Göklan Tür | Kemal Oflazer | Teruko Mitamura | Eric H. Nyberg, 3rd
Proceedings of the Third Conference of the Association for Machine Translation in the Americas: Technical Papers

This paper describes the integration of a Turkish generation system with the KANT knowledge-based machine translation system to produce a prototype English-Turkish interlingua-based machine translation system. These two independently constructed systems were successfully integrated within a period of two months, through development of a module which maps KANT interlingua expressions to Turkish syntactic structures. The combined system is able to translate completely and correctly 44 of 52 benchmark sentences in the domain of broadcast news captions. This study is the first known application of knowledge-based machine translation from English to Turkish, and our initial results show promise for future development.


pdf bib
A Real-Time MT System for Translating Broadcast Captions
Eric Nyberg | Teruko Mitamura
Proceedings of Machine Translation Summit VI: Papers

This presentation demonstrates a new multi-engine machine translation system, which combines knowledge-based and example-based machine translation strategies for real-time translation of business news captions from English to German.


Coping With Ambiguity in a Large-Scale Machine Translation System
Kathryn L. Baker | Alexander M. Franz | Pamela W. Jordan | Teruko Mitamura | Eric H. Nyberg
COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics

Evaluation Metrics for Knowledge-Based Machine Translation
Eric H. Nyberg, 3rd | Teruko Mitamura | Jaime G. Carbonell
COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics

KANT: Knowledge-Based, Accurate Natural Language Translation
Teruko Mitamura | Eric Nyberg | Jaime Carbonell
Proceedings of the First Conference of the Association for Machine Translation in the Americas


The KANT System: Fast, Accurate, High-Quality Translation in Practical Domains
Eric H. Nyberg III | Teruko Mitamura
COLING 1992 Volume 3: The 14th International Conference on Computational Linguistics

Hierarchical Lexical Structure and Interpretive Mapping in Machine Translation
Teruko Mitamura | Eric H. Nyberg III
COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics


An Efficient Interlingua Translation System for Multi-lingual Document Production
Teruko Mitamura | Eric H. Nyberg | Jaime G. Carbonell
Proceedings of Machine Translation Summit III: Papers

Knowledge-based interlingual machine translation systems produce semantically accurate translations, but typically require massive knowledge acquisition. This paper describes KANT, a system that reduces this requirement to produce practical, scalable, and accurate KBMT applications. First, the set of requirements is discussed, then the full KANT architecture is illustrated, and finally results from a fully implemented prototype are presented.


Lexical realization in natural language generation
Sergei Nirenburg | Rita McCardell | Eric Nyberg | Scott Huffman | Edward Kernschaft | Irene Nirenburg
Proceedings of the Second Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages