@inproceedings{bakarov-2018-effect,
    title = "The Effect of Unobserved Word-Context Co-occurrences on a {V}ector{M}ixture Approach for Compositional Distributional Semantics",
    author = "Bakarov, Amir",
    booktitle = "Proceedings of the Third International Conference on Computational Linguistics in Bulgaria (CLIB 2018)",
    month = may,
    year = "2018",
    address = "Sofia, Bulgaria",
    publisher = "Department of Computational Linguistics, Institute for Bulgarian Language, Bulgarian Academy of Sciences",
    url = "https://preview.aclanthology.org/fix-sig-urls/2018.clib-1.19/",
    pages = "153--161",
    abstract = "Swivel (Submatrix-WIse Vector Embedding Learner) is a distributional semantic model based on counting point-wise mutual information values, capable of capturing word-context co-occurrences in the PMI matrix that were not noted in the training corpus. This model outperforms mainstream word embedding training algorithms such as Continuous Bag-of-Words, GloVe and Skip-Gram in word similarity and word analogy tasks. But the properness of these intrinsic tasks could be questioned, and it is unclear if the ability to count unobservable word-context co-occurrences could also be helpful for downstream tasks. In this work we propose a comparison of Word2Vec and Swivel for two downstream tasks based on natural language sentence matching: the paraphrase detection task and the textual entailment task. As a result, we reveal that Swivel outperforms Word2Vec in both cases, but the difference is minuscule. We can conclude, that the ability to learn embeddings for rarely co-occurring words is not so crucial for downstream tasks."
}