@inproceedings{komiya-shinnou-2018-investigating,
    title = "Investigating Effective Parameters for Fine-tuning of Word Embeddings Using Only a Small Corpus",
    author = "Komiya, Kanako  and
      Shinnou, Hiroyuki",
    editor = "Haffari, Reza  and
      Cherry, Colin  and
      Foster, George  and
      Khadivi, Shahram  and
      Salehi, Bahar",
    booktitle = "Proceedings of the Workshop on Deep Learning Approaches for Low-Resource {NLP}",
    month = jul,
    year = "2018",
    address = "Melbourne",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/W18-3408/",
    doi = "10.18653/v1/W18-3408",
    pages = "60--67",
    abstract = "Fine-tuning is a popular method to achieve better performance when only a small target corpus is available. However, it requires tuning of a number of metaparameters and thus it might carry risk of adverse effect when inappropriate metaparameters are used. Therefore, we investigate effective parameters for fine-tuning when only a small target corpus is available. In the current study, we target at improving Japanese word embeddings created from a huge corpus. First, we demonstrate that even the word embeddings created from the huge corpus are affected by domain shift. After that, we investigate effective parameters for fine-tuning of the word embeddings using a small target corpus. We used perplexity of a language model obtained from a Long Short-Term Memory network to assess the word embeddings input into the network. The experiments revealed that fine-tuning sometimes give adverse effect when only a small target corpus is used and batch size is the most important parameter for fine-tuning. In addition, we confirmed that effect of fine-tuning is higher when size of a target corpus was larger."
}