Abstract
Word embeddings have undoubtedly revolutionized NLP. However, pretrained embeddings do not always work for a specific task (or set of tasks), particularly in limited resource setups. We introduce a simple yet effective, self-supervised post-processing method that constructs task-specialized word representations by picking from a menu of reconstructing transformations to yield improved end-task performance (MORTY). The method is complementary to recent state-of-the-art approaches to inductive transfer via fine-tuning, and forgoes costly model architectures and annotation. We evaluate MORTY on a broad range of setups, including different word embedding methods, corpus sizes and end-task semantics. Finally, we provide a surprisingly simple recipe to obtain specialized embeddings that better fit end-tasks.- Anthology ID:
- W19-4307
- Volume:
- Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)
- Month:
- August
- Year:
- 2019
- Address:
- Florence, Italy
- Venue:
- RepL4NLP
- SIG:
- SIGREP
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 49–54
- Language:
- URL:
- https://aclanthology.org/W19-4307
- DOI:
- 10.18653/v1/W19-4307
- Cite (ACL):
- Nils Rethmeier and Barbara Plank. 2019. MoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 49–54, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- MoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding (Rethmeier & Plank, RepL4NLP 2019)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/W19-4307.pdf
- Code
- NilsRethmeier/MoRTy