Goodwill Hunting: Analyzing and Repurposing Off-the-Shelf Named Entity Linking Systems
Karan Goel, Laurel Orr, Nazneen Fatema Rajani, Jesse Vig, Christopher Ré
Abstract
Named entity linking (NEL) or mapping “strings” to “things” in a knowledge base is a fundamental preprocessing step in systems that require knowledge of entities such as information extraction and question answering. In this work, we lay out and investigate two challenges faced by individuals or organizations building NEL systems. Can they directly use an off-the-shelf system? If not, how easily can such a system be repurposed for their use case? First, we conduct a study of off-the-shelf commercial and academic NEL systems. We find that most systems struggle to link rare entities, with commercial solutions lagging their academic counterparts by 10%+. Second, for a use case where the NEL model is used in a sports question-answering (QA) system, we investigate how to close the loop in our analysis by repurposing the best off-the-shelf model (Bootleg) to correct sport-related errors. We show how tailoring a simple technique for patching models using weak labeling can provide a 25% absolute improvement in accuracy of sport-related errors.- Anthology ID:
- 2021.naacl-industry.26
- Volume:
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers
- Month:
- June
- Year:
- 2021
- Address:
- Online
- Editors:
- Young-bum Kim, Yunyao Li, Owen Rambow
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 205–213
- Language:
- URL:
- https://aclanthology.org/2021.naacl-industry.26
- DOI:
- 10.18653/v1/2021.naacl-industry.26
- Cite (ACL):
- Karan Goel, Laurel Orr, Nazneen Fatema Rajani, Jesse Vig, and Christopher Ré. 2021. Goodwill Hunting: Analyzing and Repurposing Off-the-Shelf Named Entity Linking Systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, pages 205–213, Online. Association for Computational Linguistics.
- Cite (Informal):
- Goodwill Hunting: Analyzing and Repurposing Off-the-Shelf Named Entity Linking Systems (Goel et al., NAACL 2021)
- PDF:
- https://preview.aclanthology.org/naacl-24-ws-corrections/2021.naacl-industry.26.pdf