A Model for Context-Based Evaluation of Language Processing Systems and its Application to Machine Translation Evaluation

Andrei Popescu-Belis, Paula Estrella, Margaret King, Nancy Underwood

Abstract
In this paper, we propose a formal framework that takes into account the influence of the intended context of use of an NLP system on the procedure and the metrics used to evaluate the system. We introduce in particular the notion of a context-dependent quality model and explain how it can be adapted to a given context of use. More specifically, we define vector-space representations of contexts of use and of quality models, which are connected by a generic contextual quality model (GCQM). For each domain, experts in evaluation are needed to build a GCQM based on analytic knowledge and on previous evaluations, using the mechanism proposed here. The main inspiration source for this work is the FEMTI framework for the evaluation of machine translation, which implements partly the present model, and which is described briefly along with insights from other domains.
Anthology ID:
L06-1091
Volume:
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)
Month:
May
Year:
2006
Address:
Genoa, Italy
Editors:
Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, Daniel Tapias
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2006/pdf/171_pdf.pdf
DOI:
Bibkey:
Cite (ACL):
Andrei Popescu-Belis, Paula Estrella, Margaret King, and Nancy Underwood. 2006. A Model for Context-Based Evaluation of Language Processing Systems and its Application to Machine Translation Evaluation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).
Cite (Informal):
A Model for Context-Based Evaluation of Language Processing Systems and its Application to Machine Translation Evaluation (Popescu-Belis et al., LREC 2006)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2006/pdf/171_pdf.pdf
PDF Search