Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer
Chris van der Lee, Bart Verduijn, Emiel Krahmer, Sander Wubben
Abstract
We present an evaluation of PASS, a data-to-text system that generates Dutch soccer reports from match statistics which are automatically tailored towards fans of one club or the other. The evaluation in this paper consists of two studies. An intrinsic human-based evaluation of the system’s output is described in the first study. In this study it was found that compared to human-written texts, computer-generated texts were rated slightly lower on style-related text components (fluency and clarity) and slightly higher in terms of the correctness of given information. Furthermore, results from the first study showed that tailoring was accurately recognized in most cases, and that participants struggled with correctly identifying whether a text was written by a human or computer. The second study investigated if tailoring affects perceived text quality, for which no results were garnered. This lack of results might be due to negative preconceptions about computer-generated texts which were found in the first study.- Anthology ID:
- C18-1082
- Volume:
- Proceedings of the 27th International Conference on Computational Linguistics
- Month:
- August
- Year:
- 2018
- Address:
- Santa Fe, New Mexico, USA
- Editors:
- Emily M. Bender, Leon Derczynski, Pierre Isabelle
- Venue:
- COLING
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 962–972
- Language:
- URL:
- https://aclanthology.org/C18-1082
- DOI:
- Cite (ACL):
- Chris van der Lee, Bart Verduijn, Emiel Krahmer, and Sander Wubben. 2018. Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer. In Proceedings of the 27th International Conference on Computational Linguistics, pages 962–972, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
- Cite (Informal):
- Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer (van der Lee et al., COLING 2018)
- PDF:
- https://preview.aclanthology.org/ingest-2024-clasp/C18-1082.pdf