@inproceedings{liu-etal-2024-beyond-human,
    title = "Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data",
    author = "Liu, Zhongtao  and
      Riley, Parker  and
      Deutsch, Daniel  and
      Lui, Alison  and
      Niu, Mengmeng  and
      Shah, Apurva  and
      Freitag, Markus",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Ninth Conference on Machine Translation",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2024.wmt-1.110/",
    doi = "10.18653/v1/2024.wmt-1.110",
    pages = "1095--1106",
    abstract = "Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. This study presents a comprehensive investigation of 11 approaches for acquiring translation data, including human-only, machine-only, and hybrid approaches. Our findings demonstrate that human-machine collaboration can match or even exceed the quality of human-only translations, while being more cost-efficient. Error analysis reveals the complementary strengths between human and machine contributions, highlighting the effectiveness of collaborative methods. Cost analysis further demonstrates the economic benefits of human-machine collaboration methods, with some approaches achieving top-tier quality at around 60{\%} of the cost of traditional methods. We release a publicly available dataset containing nearly 18,000 segments of varying translation quality with corresponding human ratings to facilitate future research."
}