Abstract
The high performance of large pretrained language models (LLMs) such as BERT on NLP tasks has prompted questions about BERT’s linguistic capabilities, and how they differ from humans’. In this paper, we approach this question by examining BERT’s knowledge of lexical semantic relations. We focus on hypernymy, the “is-a” relation that relates a word to a superordinate category. We use a prompting methodology to simply ask BERT what the hypernym of a given word is. We find that, in a setting where all hypernyms are guessable via prompting, BERT knows hypernyms with up to 57% accuracy. Moreover, BERT with prompting outperforms other unsupervised models for hypernym discovery even in an unconstrained scenario. However, BERT’s predictions and performance on a dataset containing uncommon hyponyms and hypernyms indicate that its knowledge of hypernymy is still limited.- Anthology ID:
- 2021.blackboxnlp-1.20
- Volume:
- Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP
- Month:
- November
- Year:
- 2021
- Address:
- Punta Cana, Dominican Republic
- Venue:
- BlackboxNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 275–282
- Language:
- URL:
- https://aclanthology.org/2021.blackboxnlp-1.20
- DOI:
- 10.18653/v1/2021.blackboxnlp-1.20
- Cite (ACL):
- Michael Hanna and David Mareček. 2021. Analyzing BERT’s Knowledge of Hypernymy via Prompting. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 275–282, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Analyzing BERT’s Knowledge of Hypernymy via Prompting (Hanna & Mareček, BlackboxNLP 2021)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2021.blackboxnlp-1.20.pdf
- Data
- SemEval-2018 Task 9: Hypernym Discovery