What’s in a Name? Are BERT Named Entity Representations just as Good for any other Name?
Sriram Balasubramanian, Naman Jain, Gaurav Jindal, Abhijeet Awasthi, Sunita Sarawagi
Abstract
We evaluate named entity representations of BERT-based NLP models by investigating their robustness to replacements from the same typed class in the input. We highlight that on several tasks while such perturbations are natural, state of the art trained models are surprisingly brittle. The brittleness continues even with the recent entity-aware BERT models. We also try to discern the cause of this non-robustness, considering factors such as tokenization and frequency of occurrence. Then we provide a simple method that ensembles predictions from multiple replacements while jointly modeling the uncertainty of type annotations and label predictions. Experiments on three NLP tasks shows that our method enhances robustness and increases accuracy on both natural and adversarial datasets.- Anthology ID:
- 2020.repl4nlp-1.24
- Volume:
- Proceedings of the 5th Workshop on Representation Learning for NLP
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Spandana Gella, Johannes Welbl, Marek Rei, Fabio Petroni, Patrick Lewis, Emma Strubell, Minjoon Seo, Hannaneh Hajishirzi
- Venue:
- RepL4NLP
- SIG:
- SIGREP
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 205–214
- Language:
- URL:
- https://aclanthology.org/2020.repl4nlp-1.24
- DOI:
- 10.18653/v1/2020.repl4nlp-1.24
- Cite (ACL):
- Sriram Balasubramanian, Naman Jain, Gaurav Jindal, Abhijeet Awasthi, and Sunita Sarawagi. 2020. What’s in a Name? Are BERT Named Entity Representations just as Good for any other Name?. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 205–214, Online. Association for Computational Linguistics.
- Cite (Informal):
- What’s in a Name? Are BERT Named Entity Representations just as Good for any other Name? (Balasubramanian et al., RepL4NLP 2020)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2020.repl4nlp-1.24.pdf
- Data
- SQuAD