Cognitive demand and cognitive effort in post-editing

Isabel Lacruz, Michael Denkowski, Alon Lavie

Abstract
The pause to word ratio, the number of pauses per word in a post-edited MT segment, is an indicator of cognitive effort in post-editing (Lacruz and Shreve, 2014). We investigate how low the pause threshold can reasonably be taken, and we propose that 300 ms is a good choice, as pioneered by Schilperoord (1996). We then seek to identify a good measure of the cognitive demand imposed by MT output on the post-editor, as opposed to the cognitive effort actually exerted by the post-editor during post-editing. Measuring cognitive demand is closely related to measuring MT utility, the MT quality as perceived by the post-editor. HTER, an extrinsic edit to word ratio that does not necessarily correspond to actual edits per word performed by the post-editor, is a well-established measure of MT quality, but it does not comprehensively capture cognitive demand (Koponen, 2012). We investigate intrinsic measures of MT quality, and so of cognitive demand, through edited-error to word metrics. We find that the transfer-error to word ratio predicts cognitive effort better than mechanical-error to word ratio (Koby and Champe, 2013). We identify specific categories of cognitively challenging MT errors whose error to word ratios correlate well with cognitive effort.
Anthology ID:
2014.amta-wptp.6
Volume:
Proceedings of the 11th Conference of the Association for Machine Translation in the Americas
Month:
October 22-26
Year:
2014
Address:
Vancouver, Canada
Editors:
Sharon O'Brien, Michel Simard, Lucia Specia
Venue:
AMTA
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
73–84
Language:
URL:
https://aclanthology.org/2014.amta-wptp.6
DOI:
Bibkey:
Cite (ACL):
Isabel Lacruz, Michael Denkowski, and Alon Lavie. 2014. Cognitive demand and cognitive effort in post-editing. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, pages 73–84, Vancouver, Canada. Association for Machine Translation in the Americas.
Cite (Informal):
Cognitive demand and cognitive effort in post-editing (Lacruz et al., AMTA 2014)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2014.amta-wptp.6.pdf
PDF Search