Abstract
The current most popular method for automatic Natural Language Generation (NLG) evaluation is comparing generated text with human-written reference sentences using a metrics system, which has drawbacks around reliability and scalability. We draw inspiration from second language (L2) assessment and extract a set of linguistic features to predict human judgments of sentence naturalness. Our experiment using a small dataset showed that the feature-based approach yields promising results, with the added potential of providing interpretability into the source of the problems.- Anthology ID:
- W18-6512
- Volume:
- Proceedings of the 11th International Conference on Natural Language Generation
- Month:
- November
- Year:
- 2018
- Address:
- Tilburg University, The Netherlands
- Editors:
- Emiel Krahmer, Albert Gatt, Martijn Goudbeek
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 109–118
- Language:
- URL:
- https://aclanthology.org/W18-6512
- DOI:
- 10.18653/v1/W18-6512
- Cite (ACL):
- Isabel Groves, Ye Tian, and Ioannis Douratsos. 2018. Treat the system like a human student: Automatic naturalness evaluation of generated text without reference texts. In Proceedings of the 11th International Conference on Natural Language Generation, pages 109–118, Tilburg University, The Netherlands. Association for Computational Linguistics.
- Cite (Informal):
- Treat the system like a human student: Automatic naturalness evaluation of generated text without reference texts (Groves et al., INLG 2018)
- PDF:
- https://preview.aclanthology.org/landing_page/W18-6512.pdf