Perceptual Quality Dimensions of Machine-Generated Text with a Focus on Machine Translation
Vivien Macketanz, Babak Naderi, Steven Schmidt, Sebastian Möller
Abstract
The quality of machine-generated text is a complex construct consisting of various aspects and dimensions. We present a study that aims to uncover relevant perceptual quality dimensions for one type of machine-generated text, that is, Machine Translation. We conducted a crowdsourcing survey in the style of a Semantic Differential to collect attribute ratings for German MT outputs. An Exploratory Factor Analysis revealed the underlying perceptual dimensions. As a result, we extracted four factors that operate as relevant dimensions for the Quality of Experience of MT outputs: precision, complexity, grammaticality, and transparency.- Anthology ID:
- 2022.humeval-1.3
- Volume:
- Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Anya Belz, Maja Popović, Ehud Reiter, Anastasia Shimorina
- Venue:
- HumEval
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 24–31
- Language:
- URL:
- https://aclanthology.org/2022.humeval-1.3
- DOI:
- 10.18653/v1/2022.humeval-1.3
- Cite (ACL):
- Vivien Macketanz, Babak Naderi, Steven Schmidt, and Sebastian Möller. 2022. Perceptual Quality Dimensions of Machine-Generated Text with a Focus on Machine Translation. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 24–31, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Perceptual Quality Dimensions of Machine-Generated Text with a Focus on Machine Translation (Macketanz et al., HumEval 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2022.humeval-1.3.pdf
- Code
- dfki-nlp/textq