@inproceedings{peinelt-etal-2021-gibert-enhancing,
    title = "{G}i{BERT}: Enhancing {BERT} with Linguistic Information using a Lightweight Gated Injection Method",
    author = "Peinelt, Nicole  and
      Rei, Marek  and
      Liakata, Maria",
    editor = "Moens, Marie-Francine  and
      Huang, Xuanjing  and
      Specia, Lucia  and
      Yih, Scott Wen-tau",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2021.findings-emnlp.200/",
    doi = "10.18653/v1/2021.findings-emnlp.200",
    pages = "2322--2336",
    abstract = "Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words {--} either through masking or next sentence prediction {--} and has no knowledge of lexical, syntactic or semantic information beyond what it picks up through unsupervised pre-training. We propose a novel method to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. When injecting counter-fitted and dependency-based embeddings, the performance improvements on multiple semantic similarity datasets indicate that such information is beneficial and currently missing from the original model. Our qualitative analysis shows that counter-fitted embedding injection is particularly beneficial, with notable improvements on examples that require synonym resolution."
}