Abstract
This paper addresses the problem of representation learning. Using an autoencoder framework, we propose and evaluate several loss functions that can be used as an alternative to the commonly used cross-entropy reconstruction loss. The proposed loss functions use similarities between words in the embedding space, and can be used to train any neural model for text generation. We show that the introduced loss functions amplify semantic diversity of reconstructed sentences, while preserving the original meaning of the input. We test the derived autoencoder-generated representations on paraphrase detection and language inference tasks and demonstrate performance improvement compared to the traditional cross-entropy loss.- Anthology ID:
- D18-1525
- Volume:
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Month:
- October-November
- Year:
- 2018
- Address:
- Brussels, Belgium
- Editors:
- Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
- Venue:
- EMNLP
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4875–4880
- Language:
- URL:
- https://aclanthology.org/D18-1525
- DOI:
- 10.18653/v1/D18-1525
- Cite (ACL):
- Olga Kovaleva, Anna Rumshisky, and Alexey Romanov. 2018. Similarity-Based Reconstruction Loss for Meaning Representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4875–4880, Brussels, Belgium. Association for Computational Linguistics.
- Cite (Informal):
- Similarity-Based Reconstruction Loss for Meaning Representation (Kovaleva et al., EMNLP 2018)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/D18-1525.pdf
- Data
- SNLI