MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling
Parishad BehnamGhader, Hossein Zakerinia, Mahdieh Soleymani Baghshah
Abstract
Pre-trained models like Bidirectional Encoder Representations from Transformers (BERT), have recently made a big leap forward in Natural Language Processing (NLP) tasks. However, there are still some shortcomings in the Masked Language Modeling (MLM) task performed by these models. In this paper, we first introduce a multi-graph including different types of relations between words. Then, we propose Multi-Graph augmented BERT (MG-BERT) model that is based on BERT. MG-BERT embeds tokens while taking advantage of a static multi-graph containing global word co-occurrences in the text corpus beside global real-world facts about words in knowledge graphs. The proposed model also employs a dynamic sentence graph to capture local context effectively. Experimental results demonstrate that our model can considerably enhance the performance in the MLM task.- Anthology ID:
- 2021.textgraphs-1.12
- Volume:
- Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)
- Month:
- June
- Year:
- 2021
- Address:
- Mexico City, Mexico
- Editors:
- Alexander Panchenko, Fragkiskos D. Malliaros, Varvara Logacheva, Abhik Jana, Dmitry Ustalov, Peter Jansen
- Venue:
- TextGraphs
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 125–131
- Language:
- URL:
- https://aclanthology.org/2021.textgraphs-1.12
- DOI:
- 10.18653/v1/2021.textgraphs-1.12
- Cite (ACL):
- Parishad BehnamGhader, Hossein Zakerinia, and Mahdieh Soleymani Baghshah. 2021. MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling. In Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15), pages 125–131, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling (BehnamGhader et al., TextGraphs 2021)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2021.textgraphs-1.12.pdf
- Data
- CoLA, SST, SST-2