<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bosc-vincent-2020-sequence">
    <titleInfo>
        <title>Do sequence-to-sequence VAEs learn global features of sentences?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tom</namePart>
        <namePart type="family">Bosc</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pascal</namePart>
        <namePart type="family">Vincent</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-nov</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Autoregressive language models are powerful and relatively easy to train. However, these models are usually trained without explicit conditioning labels and do not offer easy ways to control global aspects such as sentiment or topic during generation. Bowman &amp; al. 2016 adapted the Variational Autoencoder (VAE) for natural language with the sequence-to-sequence architecture and claimed that the latent vector was able to capture such global features in an unsupervised manner. We question this claim. We measure which words benefit most from the latent information by decomposing the reconstruction loss per position in the sentence. Using this method, we find that VAEs are prone to memorizing the first words and the sentence length, producing local features of limited usefulness. To alleviate this, we investigate alternative architectures based on bag-of-words assumptions and language model pretraining. These variants learn latent variables that are more global, i.e., more predictive of topic or sentiment labels. Moreover, using reconstructions, we observe that they decrease memorization: the first word and the sentence length are not recovered as accurately than with the baselines, consequently yielding more diverse reconstructions.</abstract>
    <identifier type="citekey">bosc-vincent-2020-sequence</identifier>
    <identifier type="doi">10.18653/v1/2020.emnlp-main.350</identifier>
    <location>
        <url>https://aclanthology.org/2020.emnlp-main.350</url>
    </location>
    <part>
        <date>2020-nov</date>
        <extent unit="page">
            <start>4296</start>
            <end>4318</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Do sequence-to-sequence VAEs learn global features of sentences?
%A Bosc, Tom
%A Vincent, Pascal
%S Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
%D 2020
%8 nov
%I Association for Computational Linguistics
%C Online
%F bosc-vincent-2020-sequence
%X Autoregressive language models are powerful and relatively easy to train. However, these models are usually trained without explicit conditioning labels and do not offer easy ways to control global aspects such as sentiment or topic during generation. Bowman & al. 2016 adapted the Variational Autoencoder (VAE) for natural language with the sequence-to-sequence architecture and claimed that the latent vector was able to capture such global features in an unsupervised manner. We question this claim. We measure which words benefit most from the latent information by decomposing the reconstruction loss per position in the sentence. Using this method, we find that VAEs are prone to memorizing the first words and the sentence length, producing local features of limited usefulness. To alleviate this, we investigate alternative architectures based on bag-of-words assumptions and language model pretraining. These variants learn latent variables that are more global, i.e., more predictive of topic or sentiment labels. Moreover, using reconstructions, we observe that they decrease memorization: the first word and the sentence length are not recovered as accurately than with the baselines, consequently yielding more diverse reconstructions.
%R 10.18653/v1/2020.emnlp-main.350
%U https://aclanthology.org/2020.emnlp-main.350
%U https://doi.org/10.18653/v1/2020.emnlp-main.350
%P 4296-4318

Markdown (Informal)

[Do sequence-to-sequence VAEs learn global features of sentences?](https://aclanthology.org/2020.emnlp-main.350) (Bosc & Vincent, EMNLP 2020)

• Do sequence-to-sequence VAEs learn global features of sentences? (Bosc & Vincent, EMNLP 2020)

ACL

• Tom Bosc and Pascal Vincent. 2020. Do sequence-to-sequence VAEs learn global features of sentences?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4296–4318, Online. Association for Computational Linguistics.