@inproceedings{carl-etal-2024-using,
    title = "Using Machine Learning to Validate a Novel Taxonomy of Phenomenal Translation States",
    author = "Carl, Michael  and
      Lu, Sheng  and
      Al-Ramadan, Ali",
    editor = "Scarton, Carolina  and
      Prescott, Charlotte  and
      Bayliss, Chris  and
      Oakley, Chris  and
      Wright, Joanna  and
      Wrigley, Stuart  and
      Song, Xingyi  and
      Gow-Smith, Edward  and
      Bawden, Rachel  and
      S{\'a}nchez-Cartagena, V{\'i}ctor M  and
      Cadwell, Patrick  and
      Lapshinova-Koltunski, Ekaterina  and
      Cabarr{\~a}o, Vera  and
      Chatzitheodorou, Konstantinos  and
      Nurminen, Mary  and
      Kanojia, Diptesh  and
      Moniz, Helena",
    booktitle = "Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1)",
    month = jun,
    year = "2024",
    address = "Sheffield, UK",
    publisher = "European Association for Machine Translation (EAMT)",
    url = "https://preview.aclanthology.org/ingest-emnlp/2024.eamt-1.40/",
    pages = "480--491",
    abstract = "We report an experiment in which we use machine learning to validate the empirical objectivity of a novel annotation taxonomy for behavioral translation data. The HOF taxonomy defines three translation states according to which a human translator can be in a state of Orientation (O), Hesitation (H) or in a Flow state (F). We aim at validating the taxonomy based on a manually annotated dataset that consists of six English-Spanish translation sessions (approx 900 words) and 1813 HOF-annotated Activity Units (AUs). Two annotators annotated the data and obtain high average inter-annotator accuracy 0.76 (kappa 0.88). We train two classifiers, a Multi-layer Perceptron (MLP) and a Random Forest (RF) on the annotated data and tested on held-out data. The classifiers perform well on the annotated data and thus confirm the epistemological objectivity of the annotation taxonomy. Interestingly, inter-classifier accuracy scores are higher than between the two human annotators."
}