<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhou-etal-2021-improving-sequence">
    <titleInfo>
        <title>Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Wangchunshu</namePart>
        <namePart type="family">Zhou</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tao</namePart>
        <namePart type="family">Ge</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Canwen</namePart>
        <namePart type="family">Xu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ke</namePart>
        <namePart type="family">Xu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Furu</namePart>
        <namePart type="family">Wei</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-nov</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online and Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper, we propose Sequence Span Rewriting (SSR), a self-supervised task for sequence-to-sequence (Seq2Seq) pre-training. SSR learns to refine the machine-generated imperfect text spans into ground truth text. SSR provides more fine-grained and informative supervision in addition to the original text-infilling objective. Compared to the prevalent text infilling objectives for Seq2Seq pre-training, SSR is naturally more consistent with many downstream generation tasks that require sentence rewriting (e.g., text summarization, question generation, grammatical error correction, and paraphrase generation). We conduct extensive experiments by using SSR to improve the typical Seq2Seq pre-trained model T5 in a continual pre-training setting and show substantial improvements over T5 on various natural language generation tasks.</abstract>
    <identifier type="citekey">zhou-etal-2021-improving-sequence</identifier>
    <identifier type="doi">10.18653/v1/2021.emnlp-main.45</identifier>
    <location>
        <url>https://aclanthology.org/2021.emnlp-main.45</url>
    </location>
    <part>
        <date>2021-nov</date>
        <extent unit="page">
            <start>571</start>
            <end>582</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
%A Zhou, Wangchunshu
%A Ge, Tao
%A Xu, Canwen
%A Xu, Ke
%A Wei, Furu
%S Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
%D 2021
%8 nov
%I Association for Computational Linguistics
%C Online and Punta Cana, Dominican Republic
%F zhou-etal-2021-improving-sequence
%X In this paper, we propose Sequence Span Rewriting (SSR), a self-supervised task for sequence-to-sequence (Seq2Seq) pre-training. SSR learns to refine the machine-generated imperfect text spans into ground truth text. SSR provides more fine-grained and informative supervision in addition to the original text-infilling objective. Compared to the prevalent text infilling objectives for Seq2Seq pre-training, SSR is naturally more consistent with many downstream generation tasks that require sentence rewriting (e.g., text summarization, question generation, grammatical error correction, and paraphrase generation). We conduct extensive experiments by using SSR to improve the typical Seq2Seq pre-trained model T5 in a continual pre-training setting and show substantial improvements over T5 on various natural language generation tasks.
%R 10.18653/v1/2021.emnlp-main.45
%U https://aclanthology.org/2021.emnlp-main.45
%U https://doi.org/10.18653/v1/2021.emnlp-main.45
%P 571-582

Markdown (Informal)

[Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting](https://aclanthology.org/2021.emnlp-main.45) (Zhou et al., EMNLP 2021)

• Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting (Zhou et al., EMNLP 2021)

ACL

• Wangchunshu Zhou, Tao Ge, Canwen Xu, Ke Xu, and Furu Wei. 2021. Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 571–582, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.