@inproceedings{el-kishky-etal-2018-entropy,
    title = "Entropy-Based Subword Mining with an Application to Word Embeddings",
    author = "El-Kishky, Ahmed  and
      Xu, Frank  and
      Zhang, Aston  and
      Macke, Stephen  and
      Han, Jiawei",
    editor = {Faruqui, Manaal  and
      Sch{\"u}tze, Hinrich  and
      Trancoso, Isabel  and
      Tsvetkov, Yulia  and
      Yaghoobzadeh, Yadollah},
    booktitle = "Proceedings of the Second Workshop on Subword/Character {LE}vel Models",
    month = jun,
    year = "2018",
    address = "New Orleans",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/W18-1202",
    doi = "10.18653/v1/W18-1202",
    pages = "12--21",
    abstract = "Recent literature has shown a wide variety of benefits to mapping traditional one-hot representations of words and phrases to lower-dimensional real-valued vectors known as word embeddings. Traditionally, most word embedding algorithms treat each word as the finest meaningful semantic granularity and perform embedding by learning distinct embedding vectors for each word. Contrary to this line of thought, technical domains such as scientific and medical literature compose words from subword structures such as prefixes, suffixes, and root-words as well as compound words. Treating individual words as the finest-granularity unit discards meaningful shared semantic structure between words sharing substructures. This not only leads to poor embeddings for text corpora that have long-tail distributions, but also heuristic methods for handling out-of-vocabulary words. In this paper we propose SubwordMine, an entropy-based subword mining algorithm that is fast, unsupervised, and fully data-driven. We show that this allows for great cross-domain performance in identifying semantically meaningful subwords. We then investigate utilizing the mined subwords within the FastText embedding model and compare performance of the learned representations in a downstream language modeling task.",
}