<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lee-etal-2020-slm">
    <titleInfo>
        <title>SLM: Learning a Discourse Language Representation with Sentence Unshuffling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Haejun</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Drew</namePart>
        <namePart type="given">A</namePart>
        <namePart type="family">Hudson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kangwook</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Christopher</namePart>
        <namePart type="given">D</namePart>
        <namePart type="family">Manning</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-nov</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation in a fully self-supervised manner. Recent pre-training methods in NLP focus on learning either bottom or top-level language representations: contextualized word representations derived from language model objectives at one extreme and a whole sequence representation learned by order classification of two given textual segments at the other. However, these models are not directly encouraged to capture representations of intermediate-size structures that exist in natural languages such as sentences and the relationships among them. To that end, we propose a new approach to encourage learning of a contextualized sentence-level representation by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. Through experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show that this feature of our model improves the performance of the original BERT by large margins.</abstract>
    <identifier type="citekey">lee-etal-2020-slm</identifier>
    <identifier type="doi">10.18653/v1/2020.emnlp-main.120</identifier>
    <location>
        <url>https://aclanthology.org/2020.emnlp-main.120</url>
    </location>
    <part>
        <date>2020-nov</date>
        <extent unit="page">
            <start>1551</start>
            <end>1562</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T SLM: Learning a Discourse Language Representation with Sentence Unshuffling
%A Lee, Haejun
%A Hudson, Drew A.
%A Lee, Kangwook
%A Manning, Christopher D.
%S Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
%D 2020
%8 nov
%I Association for Computational Linguistics
%C Online
%F lee-etal-2020-slm
%X We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation in a fully self-supervised manner. Recent pre-training methods in NLP focus on learning either bottom or top-level language representations: contextualized word representations derived from language model objectives at one extreme and a whole sequence representation learned by order classification of two given textual segments at the other. However, these models are not directly encouraged to capture representations of intermediate-size structures that exist in natural languages such as sentences and the relationships among them. To that end, we propose a new approach to encourage learning of a contextualized sentence-level representation by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. Through experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show that this feature of our model improves the performance of the original BERT by large margins.
%R 10.18653/v1/2020.emnlp-main.120
%U https://aclanthology.org/2020.emnlp-main.120
%U https://doi.org/10.18653/v1/2020.emnlp-main.120
%P 1551-1562

Markdown (Informal)

[SLM: Learning a Discourse Language Representation with Sentence Unshuffling](https://aclanthology.org/2020.emnlp-main.120) (Lee et al., EMNLP 2020)

• SLM: Learning a Discourse Language Representation with Sentence Unshuffling (Lee et al., EMNLP 2020)

ACL

• Haejun Lee, Drew A. Hudson, Kangwook Lee, and Christopher D. Manning. 2020. SLM: Learning a Discourse Language Representation with Sentence Unshuffling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1551–1562, Online. Association for Computational Linguistics.