Abstract
Post-editing of machine translation has become more common in recent years. This has created the need for a formal method of assessing the performance of post-editors in terms of whether they are able to produce post-edited target texts that follow project specifications. This paper proposes the use of formalized structured translation specifications (FSTS) as a basis for post-editor assessment. To determine if potential evaluators are able to reliably assess the quality of post-edited translations, an experiment used texts representing the work of five fictional post-editors. Two software applications were developed to facilitate the assessment: the Ruqual Specifications Writer, which aids in establishing post-editing project specifications; and Ruqual Rubric Viewer, which provides a graphical user interface for constructing a rubric in a machine-readable format. Seventeen non-experts rated the translation quality of each simulated post-edited text. Intraclass correlation analysis showed evidence that the evaluators were highly reliable in evaluating the performance of the post-editors. Thus, we assert that using FSTS specifications applied through the Ruqual software tools provides a useful basis for evaluating the quality of post-edited texts.- Anthology ID:
- 2012.amta-wptp.4
- Volume:
- Workshop on Post-Editing Technology and Practice
- Month:
- October 28
- Year:
- 2012
- Address:
- San Diego, California, USA
- Editors:
- Sharon O'Brien, Michel Simard, Lucia Specia
- Venue:
- AMTA
- SIG:
- Publisher:
- Association for Machine Translation in the Americas
- Note:
- Pages:
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2012.amta-wptp.4/
- DOI:
- Cite (ACL):
- Alan K. Melby, Jason Housley, Paul J. Fields, and Emily Tuioti. 2012. Reliably Assessing the Quality of Post-edited Translation Based on Formalized Structured Translation Specifications. In Workshop on Post-Editing Technology and Practice, San Diego, California, USA. Association for Machine Translation in the Americas.
- Cite (Informal):
- Reliably Assessing the Quality of Post-edited Translation Based on Formalized Structured Translation Specifications (Melby et al., AMTA 2012)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2012.amta-wptp.4.pdf