PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion
Jianhao Shen, Chenguang Wang, Ye Yuan, Jiawei Han, Heng Ji, Koushik Sen, Ming Zhang, Dawn Song
Abstract
This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a “fill-in-the-blank” task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-the-art approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters.- Anthology ID:
- 2022.findings-emnlp.281
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3833–3847
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.281
- DOI:
- Cite (ACL):
- Jianhao Shen, Chenguang Wang, Ye Yuan, Jiawei Han, Heng Ji, Koushik Sen, Ming Zhang, and Dawn Song. 2022. PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3833–3847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion (Shen et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2022.findings-emnlp.281.pdf