<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kanojia-etal-2021-pushing">
    <titleInfo>
        <title>Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Diptesh</namePart>
        <namePart type="family">Kanojia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marina</namePart>
        <namePart type="family">Fomicheva</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tharindu</namePart>
        <namePart type="family">Ranasinghe</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Frédéric</namePart>
        <namePart type="family">Blain</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Constantin</namePart>
        <namePart type="family">Orăsan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lucia</namePart>
        <namePart type="family">Specia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-nov</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Sixth Conference on Machine Translation</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Current Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning errors, thus undermining their reliability in practice. Quality Estimation (QE) is the task of automatically assessing the performance of MT systems at test time. Thus, in order to be useful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a general methodology for adversarial testing of QE for MT. First, we show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaning-preserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE systems without relying on manual quality annotation.</abstract>
    <identifier type="citekey">kanojia-etal-2021-pushing</identifier>
    <location>
        <url>https://aclanthology.org/2021.wmt-1.67</url>
    </location>
    <part>
        <date>2021-nov</date>
        <extent unit="page">
            <start>625</start>
            <end>638</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation
%A Kanojia, Diptesh
%A Fomicheva, Marina
%A Ranasinghe, Tharindu
%A Blain, Frédéric
%A Orăsan, Constantin
%A Specia, Lucia
%S Proceedings of the Sixth Conference on Machine Translation
%D 2021
%8 nov
%I Association for Computational Linguistics
%C Online
%F kanojia-etal-2021-pushing
%X Current Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning errors, thus undermining their reliability in practice. Quality Estimation (QE) is the task of automatically assessing the performance of MT systems at test time. Thus, in order to be useful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a general methodology for adversarial testing of QE for MT. First, we show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaning-preserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE systems without relying on manual quality annotation.
%U https://aclanthology.org/2021.wmt-1.67
%P 625-638

Markdown (Informal)

[Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation](https://aclanthology.org/2021.wmt-1.67) (Kanojia et al., WMT 2021)

• Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation (Kanojia et al., WMT 2021)

ACL

• Diptesh Kanojia, Marina Fomicheva, Tharindu Ranasinghe, Frédéric Blain, Constantin Orăsan, and Lucia Specia. 2021. Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 625–638, Online. Association for Computational Linguistics.