@inproceedings{shalev-etal-2021-randomized,
    title = "On Randomized Classification Layers and Their Implications in Natural Language Generation",
    author = "Shalev, Gal-Lev  and
      Shalev, Gabi  and
      Keshet, Joseph",
    editor = "Zadeh, Amir  and
      Morency, Louis-Philippe  and
      Liang, Paul Pu  and
      Ross, Candace  and
      Salakhutdinov, Ruslan  and
      Poria, Soujanya  and
      Cambria, Erik  and
      Shi, Kelly",
    booktitle = "Proceedings of the Third Workshop on Multimodal Artificial Intelligence",
    month = jun,
    year = "2021",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2021.maiworkshop-1.2/",
    doi = "10.18653/v1/2021.maiworkshop-1.2",
    pages = "6--11",
    abstract = "In natural language generation tasks, a neural language model is used for generating a sequence of words forming a sentence. The topmost weight matrix of the language model, known as the classification layer, can be viewed as a set of vectors, each representing a target word from the target dictionary. The target word vectors, along with the rest of the model parameters, are learned and updated during training. In this paper, we analyze the properties encoded in the target vectors and question the necessity of learning these vectors. We suggest to randomly draw the target vectors and set them as fixed so that no weights updates are being made during training. We show that by excluding the vectors from the optimization, the number of parameters drastically decreases with a marginal effect on the performance. We demonstrate the effectiveness of our method in image-captioning and machine-translation."
}