@inproceedings{chronis-erk-2020-bishop,
    title = "When is a bishop not like a rook? When it`s like a rabbi! Multi-prototype {BERT} embeddings for estimating semantic relationships",
    author = "Chronis, Gabriella  and
      Erk, Katrin",
    editor = "Fern{\'a}ndez, Raquel  and
      Linzen, Tal",
    booktitle = "Proceedings of the 24th Conference on Computational Natural Language Learning",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2020.conll-1.17/",
    doi = "10.18653/v1/2020.conll-1.17",
    pages = "227--244",
    abstract = "This paper investigates contextual language models, which produce token representations, as a resource for lexical semantics at the word or type level. We construct multi-prototype word embeddings from bert-base-uncased (Devlin et al., 2018). These embeddings retain contextual knowledge that is critical for some type-level tasks, while being less cumbersome and less subject to outlier effects than exemplar models. Similarity and relatedness estimation, both type-level tasks, benefit from this contextual knowledge, indicating the context-sensitivity of these processes. BERT`s token level knowledge also allows the testing of a type-level hypothesis about lexical abstractness, demonstrating the relationship between token-level phenomena and type-level concreteness ratings. Our findings provide important insight into the interpretability of BERT: layer 7 approximates semantic similarity, while the final layer (11) approximates relatedness."
}