A Simple and Effective Usage of Word Clusters for CBOW Model
Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura
Abstract
We propose a simple and effective method for incorporating word clusters into the Continuous Bag-of-Words (CBOW) model. Specifically, we propose to replace infrequent input and output words in CBOW model with their clusters. The resulting cluster-incorporated CBOW model produces embeddings of frequent words and a small amount of cluster embeddings, which will be fine-tuned in downstream tasks. We empirically show our replacing method works well on several downstream tasks. Through our analysis, we show that our method might be also useful for other similar models which produce word embeddings.- Anthology ID:
- 2020.aacl-main.10
- Volume:
- Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing
- Month:
- December
- Year:
- 2020
- Address:
- Suzhou, China
- Venue:
- AACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 80–86
- Language:
- URL:
- https://aclanthology.org/2020.aacl-main.10
- DOI:
- Cite (ACL):
- Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, and Manabu Okumura. 2020. A Simple and Effective Usage of Word Clusters for CBOW Model. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 80–86, Suzhou, China. Association for Computational Linguistics.
- Cite (Informal):
- A Simple and Effective Usage of Word Clusters for CBOW Model (Feng et al., AACL 2020)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2020.aacl-main.10.pdf
- Code
- yukunfeng/cluster-cbow