@inproceedings{rezaee-camacho-collados-2022-probing,
    title = "Probing Relational Knowledge in Language Models via Word Analogies",
    author = "Rezaee, Kiamehr  and
      Camacho-Collados, Jose",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2022.findings-emnlp.289/",
    doi = "10.18653/v1/2022.findings-emnlp.289",
    pages = "3930--3936",
    abstract = "Understanding relational knowledge plays an integral part in natural language comprehension. When it comes to pre-trained language models (PLM), prior work has been focusing on probing relational knowledge this by filling the blanks in pre-defined prompts such as {\textquotedblleft}The capital of France is {---}''. However, these probes may be affected by the co-occurrence of target relation words and entities (e.g. {\textquotedblleft}capital{\textquotedblright}, {\textquotedblleft}France{\textquotedblright} and {\textquotedblleft}Paris{\textquotedblright}) in the pre-training corpus. In this work, we extend these probing methodologies leveraging analogical proportions as a proxy to probe relational knowledge in transformer-based PLMs without directly presenting the desired relation. In particular, we analysed the ability of PLMs to understand (1) the directionality of a given relation (e.g. Paris-France is not the same as France-Paris); (2) the ability to distinguish types on a given relation (both France and Japan are countries); and (3) the relation itself (Paris is the capital of France, but not Rome). Our results show how PLMs are extremely accurate at (1) and (2), but have clear room for improvement for (3). To better understand the reasons behind this behaviour and mistakes made by PLMs, we provide an extended quantitative analysis based on relevant factors such as frequency."
}