<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="saleh-etal-2020-collective">
    <titleInfo>
        <title>Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Fahimeh</namePart>
        <namePart type="family">Saleh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Wray</namePart>
        <namePart type="family">Buntine</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gholamreza</namePart>
        <namePart type="family">Haffari</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-dec</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 28th International Conference on Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Barcelona, Spain (Online)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Scarcity of parallel sentence-pairs poses a significant hurdle for training high-quality Neural Machine Translation (NMT) models in bilingually low-resource scenarios. A standard approach is transfer learning, which involves taking a model trained on a high-resource language-pair and fine-tuning it on the data of the low-resource MT condition of interest. However, it is not clear generally which high-resource language-pair offers the best transfer learning for the target MT setting. Furthermore, different transferred models may have complementary semantic and/or syntactic strengths, hence using only one model may be sub-optimal. In this paper, we tackle this problem using knowledge distillation, where we propose to distill the knowledge of ensemble of teacher models to a single student model. As the quality of these teacher models varies, we propose an effective adaptive knowledge distillation approach to dynamically adjust the contribution of the teacher models during the distillation process. Experiments on transferring from a collection of six language pairs from IWSLT to five low-resource language-pairs from TED Talks demonstrate the effectiveness of our approach, achieving up to +0.9 BLEU score improvements compared to strong baselines.</abstract>
    <identifier type="citekey">saleh-etal-2020-collective</identifier>
    <identifier type="doi">10.18653/v1/2020.coling-main.302</identifier>
    <location>
        <url>https://aclanthology.org/2020.coling-main.302</url>
    </location>
    <part>
        <date>2020-dec</date>
        <extent unit="page">
            <start>3413</start>
            <end>3421</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation
%A Saleh, Fahimeh
%A Buntine, Wray
%A Haffari, Gholamreza
%S Proceedings of the 28th International Conference on Computational Linguistics
%D 2020
%8 dec
%I International Committee on Computational Linguistics
%C Barcelona, Spain (Online)
%F saleh-etal-2020-collective
%X Scarcity of parallel sentence-pairs poses a significant hurdle for training high-quality Neural Machine Translation (NMT) models in bilingually low-resource scenarios. A standard approach is transfer learning, which involves taking a model trained on a high-resource language-pair and fine-tuning it on the data of the low-resource MT condition of interest. However, it is not clear generally which high-resource language-pair offers the best transfer learning for the target MT setting. Furthermore, different transferred models may have complementary semantic and/or syntactic strengths, hence using only one model may be sub-optimal. In this paper, we tackle this problem using knowledge distillation, where we propose to distill the knowledge of ensemble of teacher models to a single student model. As the quality of these teacher models varies, we propose an effective adaptive knowledge distillation approach to dynamically adjust the contribution of the teacher models during the distillation process. Experiments on transferring from a collection of six language pairs from IWSLT to five low-resource language-pairs from TED Talks demonstrate the effectiveness of our approach, achieving up to +0.9 BLEU score improvements compared to strong baselines.
%R 10.18653/v1/2020.coling-main.302
%U https://aclanthology.org/2020.coling-main.302
%U https://doi.org/10.18653/v1/2020.coling-main.302
%P 3413-3421

Markdown (Informal)

[Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation](https://aclanthology.org/2020.coling-main.302) (Saleh et al., COLING 2020)

• Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation (Saleh et al., COLING 2020)

ACL

• Fahimeh Saleh, Wray Buntine, and Gholamreza Haffari. 2020. Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3413–3421, Barcelona, Spain (Online). International Committee on Computational Linguistics.