Automatic Quality Estimation for Natural Language Generation: Ranting (Jointly Rating and Ranking)

Ondřej Dušek, Karin Sevegnani, Ioannis Konstas, Verena Rieser

Abstract
We present a recurrent neural network based system for automatic quality estimation of natural language generation (NLG) outputs, which jointly learns to assign numerical ratings to individual outputs and to provide pairwise rankings of two different outputs. The latter is trained using pairwise hinge loss over scores from two copies of the rating network. We use learning to rank and synthetic data to improve the quality of ratings assigned by our system: We synthesise training pairs of distorted system outputs and train the system to rank the less distorted one higher. This leads to a 12% increase in correlation with human ratings over the previous benchmark. We also establish the state of the art on the dataset of relative rankings from the E2E NLG Challenge (Dusek et al., 2019), where synthetic data lead to a 4% accuracy increase over the base model.
Anthology ID:
W19-8644
Volume:
Proceedings of the 12th International Conference on Natural Language Generation
Month:
October–November
Year:
2019
Address:
Tokyo, Japan
Editors:
Kees van Deemter, Chenghua Lin, Hiroya Takamura
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
369–376
Language:
URL:
https://aclanthology.org/W19-8644
DOI:
10.18653/v1/W19-8644
Bibkey:
Cite (ACL):
Ondřej Dušek, Karin Sevegnani, Ioannis Konstas, and Verena Rieser. 2019. Automatic Quality Estimation for Natural Language Generation: Ranting (Jointly Rating and Ranking). In Proceedings of the 12th International Conference on Natural Language Generation, pages 369–376, Tokyo, Japan. Association for Computational Linguistics.
Cite (Informal):
Automatic Quality Estimation for Natural Language Generation: Ranting (Jointly Rating and Ranking) (Dušek et al., INLG 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-3/W19-8644.pdf
Supplementary attachment:
 W19-8644.Supplementary_Attachment.pdf
Code
 tuetschek/ratpred
PDF Search Code Supplementary attachment