A Comparative Evaluation Methodology for NLG in Interactive Systems

Helen Hastie, Anja Belz

Abstract
Interactive systems have become an increasingly important type of application for deployment of NLG technology over recent years. At present, we do not yet have commonly agreed terminology or methodology for evaluating NLG within interactive systems. In this paper, we take steps towards addressing this gap by presenting a set of principles for designing new evaluations in our comparative evaluation methodology. We start with presenting a categorisation framework, giving an overview of different categories of evaluation measures, in order to provide standard terminology for categorising existing and new evaluation techniques. Background on existing evaluation methodologies for NLG and interactive systems is presented. The comparative evaluation methodology is presented. Finally, a methodology for comparative evaluation of NLG components embedded within interactive systems is presented in terms of the comparative evaluation methodology, using a specific task for illustrative purposes.
Anthology ID:
L14-1102
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
4004–4011
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1147_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Helen Hastie and Anja Belz. 2014. A Comparative Evaluation Methodology for NLG in Interactive Systems. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 4004–4011, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
A Comparative Evaluation Methodology for NLG in Interactive Systems (Hastie & Belz, LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1147_Paper.pdf
PDF Search