@inproceedings{melby-etal-2012-reliably,
    title = "Reliably Assessing the Quality of Post-edited Translation Based on Formalized Structured Translation Specifications",
    author = "Melby, Alan K.  and
      Housley, Jason  and
      Fields, Paul J.  and
      Tuioti, Emily",
    editor = "O'Brien, Sharon  and
      Simard, Michel  and
      Specia, Lucia",
    booktitle = "Workshop on Post-Editing Technology and Practice",
    month = oct # " 28",
    year = "2012",
    address = "San Diego, California, USA",
    publisher = "Association for Machine Translation in the Americas",
    url = "https://preview.aclanthology.org/fix-sig-urls/2012.amta-wptp.4/",
    abstract = "Post-editing of machine translation has become more common in recent years. This has created the need for a formal method of assessing the performance of post-editors in terms of whether they are able to produce post-edited target texts that follow project specifications. This paper proposes the use of formalized structured translation specifications (FSTS) as a basis for post-editor assessment. To determine if potential evaluators are able to reliably assess the quality of post-edited translations, an experiment used texts representing the work of five fictional post-editors. Two software applications were developed to facilitate the assessment: the Ruqual Specifications Writer, which aids in establishing post-editing project specifications; and Ruqual Rubric Viewer, which provides a graphical user interface for constructing a rubric in a machine-readable format. Seventeen non-experts rated the translation quality of each simulated post-edited text. Intraclass correlation analysis showed evidence that the evaluators were highly reliable in evaluating the performance of the post-editors. Thus, we assert that using FSTS specifications applied through the Ruqual software tools provides a useful basis for evaluating the quality of post-edited texts."
}