Biomedical Entity Representation with Graph-Augmented Multi-Objective Transformer

Andrey Sakhovskiy, Natalia Semenova, Artur Kadurin, Elena Tutubalina

Abstract
Modern biomedical concept representations are mostly trained on synonymous concept names from a biomedical knowledge base, ignoring the inter-concept interactions and a concept’s local neighborhood in a knowledge base graph. In this paper, we introduce Biomedical Entity Representation with a Graph-Augmented Multi-Objective Transformer (BERGAMOT), which adopts the power of pre-trained language models (LMs) and graph neural networks to capture both inter-concept and intra-concept interactions from the multilingual UMLS graph. To obtain fine-grained graph representations, we introduce two additional graph-based objectives: (i) a node-level contrastive objective and (ii) the Deep Graph Infomax (DGI) loss, which maximizes the mutual information between a local subgraph and a high-level graph summary. We apply contrastive loss on textual and graph representations to make them less sensitive to surface forms and enable intermodal knowledge exchange. BERGAMOT achieves state-of-the-art results in zero-shot entity linking without task-specific supervision on 4 of 5 languages of the Mantra corpus and on 8 of 10 languages of the XL-BEL benchmark.
Anthology ID:
2024.findings-naacl.288
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4626–4643
Language:
URL:
https://aclanthology.org/2024.findings-naacl.288
DOI:
10.18653/v1/2024.findings-naacl.288
Bibkey:
Cite (ACL):
Andrey Sakhovskiy, Natalia Semenova, Artur Kadurin, and Elena Tutubalina. 2024. Biomedical Entity Representation with Graph-Augmented Multi-Objective Transformer. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4626–4643, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Biomedical Entity Representation with Graph-Augmented Multi-Objective Transformer (Sakhovskiy et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.findings-naacl.288.pdf
PDF Search