Helen Hastie

Also published as: Helen Wright Hastie


2021

pdf
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations
Miruna-Adriana Clinciu | Arash Eshghi | Helen Hastie
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

As transparency becomes key for robotics and AI, it will be necessary to evaluate the methods through which transparency is provided, including automatically generated natural language (NL) explanations. Here, we explore parallels between the generation of such explanations and the much-studied field of evaluation of Natural Language Generation (NLG). Specifically, we investigate which of the NLG evaluation measures map well to explanations. We present the ExBAN corpus: a crowd-sourced corpus of NL explanations for Bayesian Networks. We run correlations comparing human subjective ratings with NLG automatic measures. We find that embedding-based automatic NLG evaluation methods, such as BERTScore and BLEURT, have a higher correlation with human ratings, compared to word-overlap metrics, such as BLEU and ROUGE. This work has implications for Explainable AI and transparent robotic and autonomous systems.

pdf bib
Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps
Miltiadis Marios Katsakioris | Ioannis Konstas | Pierre Yves Mignotte | Helen Hastie
Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics

Robust situated dialog requires the ability to process instructions based on spatial information, which may or may not be available. We propose a model, based on LXMERT, that can extract spatial information from text instructions and attend to landmarks on OpenStreetMap (OSM) referred to in a natural language instruction. Whilst, OSM is a valuable resource, as with any open-sourced data, there is noise and variation in the names referred to on the map, as well as, variation in natural language instructions, hence the need for data-driven methods over rule-based systems. This paper demonstrates that the gold GPS location can be accurately predicted from the natural language instruction and metadata with 72% accuracy for previously seen maps and 64% for unseen maps.

2020

pdf
Towards Large-Scale Data Mining for Data-Driven Analysis of Sign Languages
Boris Mocialov | Graham Turner | Helen Hastie
Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives

Access to sign language data is far from adequate. We show that it is possible to collect the data from social networking services such as TikTok, Instagram, and YouTube by applying data filtering to enforce quality standards and by discovering patterns in the filtered data, making it easier to analyse and model. Using our data collection pipeline, we collect and examine the interpretation of songs in both the American Sign Language (ASL) and the Brazilian Sign Language (Libras). We explore their differences and similarities by looking at the co-dependence of the orientation and location phonological parameters.

pdf
CRWIZ: A Framework for Crowdsourcing Real-Time Wizard-of-Oz Dialogues
Francisco Javier Chiyah Garcia | José Lopes | Xingkun Liu | Helen Hastie
Proceedings of the Twelfth Language Resources and Evaluation Conference

Large corpora of task-based and open-domain conversational dialogues are hugely valuable in the field of data-driven dialogue systems. Crowdsourcing platforms, such as Amazon Mechanical Turk, have been an effective method for collecting such large amounts of data. However, difficulties arise when task-based dialogues require expert domain knowledge or rapid access to domain-relevant information, such as databases for tourism. This will become even more prevalent as dialogue systems become increasingly ambitious, expanding into tasks with high levels of complexity that require collaboration and forward planning, such as in our domain of emergency response. In this paper, we propose CRWIZ: a framework for collecting real-time Wizard of Oz dialogues through crowdsourcing for collaborative, complex tasks. This framework uses semi-guided dialogue to avoid interactions that breach procedures and processes only known to experts, while enabling the capture of a wide variety of interactions.

2019

pdf bib
Corpus of Multimodal Interaction for Collaborative Planning
Miltiadis Marios Katsakioris | Helen Hastie | Ioannis Konstas | Atanas Laskov
Proceedings of the Combined Workshop on Spatial Language Understanding (SpLU) and Grounded Communication for Robotics (RoboNLP)

As autonomous systems become more commonplace, we need a way to easily and naturally communicate to them our goals and collaboratively come up with a plan on how to achieve these goals. To this end, we conducted a Wizard of Oz study to gather data and investigate the way operators would collaboratively make plans via a conversational ‘planning assistant’ for remote autonomous systems. We present here a corpus of 22 dialogs from expert operators, which can be used to train such a system. Data analysis shows that multimodality is key to successful interaction, measured both quantitatively and qualitatively via user feedback.

pdf
A Survey of Explainable AI Terminology
Miruna-Adriana Clinciu | Helen Hastie
Proceedings of the 1st Workshop on Interactive Natural Language Technology for Explainable Artificial Intelligence (NL4XAI 2019)

2018

pdf
Transfer Learning for British Sign Language Modelling
Boris Mocialov | Helen Hastie | Graham Turner
Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)

Automatic speech recognition and spoken dialogue systems have made great advances through the use of deep machine learning methods. This is partly due to greater computing power but also through the large amount of data available in common languages, such as English. Conversely, research in minority languages, including sign languages, is hampered by the severe lack of data. This has led to work on transfer learning methods, whereby a model developed for one language is reused as the starting point for a model on a second language, which is less resourced. In this paper, we examine two transfer learning techniques of fine-tuning and layer substitution for language modelling of British Sign Language. Our results show improvement in perplexity when using transfer learning with standard stacked LSTM models, trained initially using a large corpus for standard English from the Penn Treebank corpus.

pdf
Explainable Autonomy: A Study of Explanation Styles for Building Clear Mental Models
Francisco Javier Chiyah Garcia | David A. Robb | Xingkun Liu | Atanas Laskov | Pedro Patron | Helen Hastie
Proceedings of the 11th International Conference on Natural Language Generation

As unmanned vehicles become more autonomous, it is important to maintain a high level of transparency regarding their behaviour and how they operate. This is particularly important in remote locations where they cannot be directly observed. Here, we describe a method for generating explanations in natural language of autonomous system behaviour and reasoning. Our method involves deriving an interpretable model of autonomy through having an expert ‘speak aloud’ and providing various levels of detail based on this model. Through an online evaluation study with operators, we show it is best to generate explanations with multiple possible reasons but tersely worded. This work has implications for designing interfaces for autonomy as well as for explainable AI and operator training.

2014

pdf
Comparing Multi-label Classification with Reinforcement Learning for Summarisation of Time-series Data
Dimitra Gkatzia | Helen Hastie | Oliver Lemon
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf bib
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)
Kallirroi Georgila | Matthew Stone | Helen Hastie | Ani Nenkova
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

pdf
The PARLANCE mobile application for interactive search in English and Mandarin
Helen Hastie | Marie-Aude Aufaure | Panos Alexopoulos | Hugues Bouchard | Catherine Breslin | Heriberto Cuayáhuitl | Nina Dethlefs | Milica Gašić | James Henderson | Oliver Lemon | Xingkun Liu | Peter Mika | Nesrine Ben Mustapha | Tim Potter | Verena Rieser | Blaise Thomson | Pirros Tsiakoulis | Yves Vanrompay | Boris Villazon-Terrazas | Majid Yazdani | Steve Young | Yanchao Yu
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

pdf
Multi-adaptive Natural Language Generation using Principal Component Regression
Dimitra Gkatzia | Helen Hastie | Oliver Lemon
Proceedings of the 8th International Natural Language Generation Conference (INLG)

pdf
Cluster-based Prediction of User Ratings for Stylistic Surface Realisation
Nina Dethlefs | Heriberto Cuayáhuitl | Helen Hastie | Verena Rieser | Oliver Lemon
Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics

pdf
Finding middle ground? Multi-objective Natural Language Generation from time-series data
Dimitra Gkatzia | Helen Hastie | Oliver Lemon
Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers

pdf
A Comparative Evaluation Methodology for NLG in Interactive Systems
Helen Hastie | Anja Belz
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

Interactive systems have become an increasingly important type of application for deployment of NLG technology over recent years. At present, we do not yet have commonly agreed terminology or methodology for evaluating NLG within interactive systems. In this paper, we take steps towards addressing this gap by presenting a set of principles for designing new evaluations in our comparative evaluation methodology. We start with presenting a categorisation framework, giving an overview of different categories of evaluation measures, in order to provide standard terminology for categorising existing and new evaluation techniques. Background on existing evaluation methodologies for NLG and interactive systems is presented. The comparative evaluation methodology is presented. Finally, a methodology for comparative evaluation of NLG components embedded within interactive systems is presented in terms of the comparative evaluation methodology, using a specific task for illustrative purposes.

2013

pdf
Generating Student Feedback from Time-Series Data Using Reinforcement Learning
Dimitra Gkatzia | Helen Hastie | Srinivasan Janarthanam | Oliver Lemon
Proceedings of the 14th European Workshop on Natural Language Generation

pdf
Demonstration of the PARLANCE system: a data-driven incremental, spoken dialogue system for interactive search
Helen Hastie | Marie-Aude Aufaure | Panos Alexopoulos | Heriberto Cuayáhuitl | Nina Dethlefs | Milica Gasic | James Henderson | Oliver Lemon | Xingkun Liu | Peter Mika | Nesrine Ben Mustapha | Verena Rieser | Blaise Thomson | Pirros Tsiakoulis | Yves Vanrompay
Proceedings of the SIGDIAL 2013 Conference

pdf
Impact of ASR N-Best Information on Bayesian Dialogue Act Recognition
Heriberto Cuayáhuitl | Nina Dethlefs | Helen Hastie | Oliver Lemon
Proceedings of the SIGDIAL 2013 Conference

pdf
Demonstration of the EmoteWizard of Oz Interface for Empathic Robotic Tutors
Shweta Bhargava | Srinivasan Janarthanam | Helen Hastie | Amol Deshmukh | Ruth Aylett | Lee Corrigan | Ginevra Castellano
Proceedings of the SIGDIAL 2013 Conference

pdf
Conditional Random Fields for Responsive Surface Realisation using Global Features
Nina Dethlefs | Helen Hastie | Heriberto Cuayáhuitl | Oliver Lemon
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

2012

pdf
Optimising Incremental Generation for Spoken Dialogue Systems: Reducing the Need for Fillers
Nina Dethlefs | Helen Hastie | Verena Rieser | Oliver Lemon
INLG 2012 Proceedings of the Seventh International Natural Language Generation Conference

pdf
Incremental Spoken Dialogue Systems: Tools and Data
Helen Hastie | Oliver Lemon | Nina Dethlefs
NAACL-HLT Workshop on Future directions and needs in the Spoken Dialog Community: Tools and Data (SDCTD 2012)

pdf
Optimising Incremental Dialogue Decisions Using Information Density for Interactive Systems
Nina Dethlefs | Helen Hastie | Verena Rieser | Oliver Lemon
Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning

2011

pdf bib
Spoken Dialog Challenge 2010: Comparison of Live and Control Test Results
Alan W Black | Susanne Burger | Alistair Conkie | Helen Hastie | Simon Keizer | Oliver Lemon | Nicolas Merigaud | Gabriel Parent | Gabriel Schubiner | Blaise Thomson | Jason D. Williams | Kai Yu | Steve Young | Maxine Eskenazi
Proceedings of the SIGDIAL 2011 Conference

pdf
“The day after the day after tomorrow?” A machine learning approach to adaptive temporal expression generation: training and evaluation with real users
Srinivasan Janarthanam | Helen Hastie | Oliver Lemon | Xingkun Liu
Proceedings of the SIGDIAL 2011 Conference

2009

pdf
Automatic Generation of Information State Update Dialogue Systems that Dynamically Create Voice XML, as Demonstrated on the iPhone
Helen Hastie | Xingkun Liu | Oliver Lemon
Proceedings of the SIGDIAL 2009 Conference

2008

pdf
“Build Your Own” Spoken Dialogue Systems: Automatically Generating ISU Dialogue Systems from Business User Resources
Oliver Lemon | Xingkun Liu | Helen Hastie
Coling 2008: Companion volume: Demonstrations

2007

pdf
WIRE: A Wearable Spoken Language Understanding System for the Military
Helen Hastie | Patrick Craven | Michael Orr
Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies

2003

pdf
The Pragmatics of Taking a Spoken Language System Out of the Laboratory
Jody J. Daniels | Helen Wright Hastie
Proceedings of the HLT-NAACL 2003 Workshop on Research Directions in Dialogue Processing

2002

pdf
What’s the Trouble: Automatically Identifying Problematic Dialogues in DARPA Communicator Dialogue Systems
Helen Wright Hastie | Rashmi Prasad | Marilyn Walker
Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics

pdf
Automatic Evaluation: Using a DATE Dialogue Act Tagger for User Satisfaction and Task Completion Prediction
Helen Wright Hastie | Rashmi Prasad | Marilyn Walker
Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02)