Abstract
Pre-trained word embeddings improve the performance of a neural model at the cost of increasing the model size. We propose to benefit from this resource without paying the cost by operating strictly at the sub-lexical level. Our approach is quite simple: before task-specific training, we first optimize sub-word parameters to reconstruct pre-trained word embeddings using various distance measures. We report interesting results on a variety of tasks: word similarity, word analogy, and part-of-speech tagging.- Anthology ID:
- W17-4119
- Volume:
- Proceedings of the First Workshop on Subword and Character Level Models in NLP
- Month:
- September
- Year:
- 2017
- Address:
- Copenhagen, Denmark
- Editors:
- Manaal Faruqui, Hinrich Schuetze, Isabel Trancoso, Yadollah Yaghoobzadeh
- Venue:
- SCLeM
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 130–135
- Language:
- URL:
- https://aclanthology.org/W17-4119
- DOI:
- 10.18653/v1/W17-4119
- Cite (ACL):
- Karl Stratos. 2017. Reconstruction of Word Embeddings from Sub-Word Parameters. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 130–135, Copenhagen, Denmark. Association for Computational Linguistics.
- Cite (Informal):
- Reconstruction of Word Embeddings from Sub-Word Parameters (Stratos, SCLeM 2017)
- PDF:
- https://preview.aclanthology.org/revert-3132-ingestion-checklist/W17-4119.pdf