SINotas: the Evaluation of a NLG Application

Roberto P. A. Araujo, Rafael L. de Oliveira, Eder M. de Novais, Thiago D. Tadeu, Daniel B. Pereira, Ivandré Paraboni


Abstract
SINotas is a data-to-text NLG application intended to produce short textual reports on students’ academic performance from a database conveying their grades, weekly attendance rates and related academic information. Although developed primarily as a testbed for Portuguese Natural Language Generation, SINotas generates reports of interest to both students keen to learn how their professors would describe their efforts, and to the professors themselves, who may benefit from an at-a-glance view of the student’s performance. In a traditional machine learning approach, SINotas uses a data-text aligned corpus as training data for decision-tree induction. The current system comprises a series of classifiers that implement major Document Planning subtasks (namely, data interpretation, content selection, within- and between-sentence structuring), and a small surface realisation grammar of Brazilian Portuguese. In this paper we focus on the evaluation work of the system, applying a number of intrinsic and user-based evaluation metrics to a collection of text reports generated from real application data.
Anthology ID:
L10-1406
Volume:
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)
Month:
May
Year:
2010
Address:
Valletta, Malta
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2010/pdf/593_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Roberto P. A. Araujo, Rafael L. de Oliveira, Eder M. de Novais, Thiago D. Tadeu, Daniel B. Pereira, and Ivandré Paraboni. 2010. SINotas: the Evaluation of a NLG Application. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).
Cite (Informal):
SINotas: the Evaluation of a NLG Application (Araujo et al., LREC 2010)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2010/pdf/593_Paper.pdf