Pre-training Entity Relation Encoder with Intra-span and Inter-span Information
Yijun Wang, Changzhi Sun, Yuanbin Wu, Junchi Yan, Peng Gao, Guotong Xie
Abstract
In this paper, we integrate span-related information into pre-trained encoder for entity relation extraction task. Instead of using general-purpose sentence encoder (e.g., existing universal pre-trained models), we introduce a span encoder and a span pair encoder to the pre-training network, which makes it easier to import intra-span and inter-span information into the pre-trained model. To learn the encoders, we devise three customized pre-training objectives from different perspectives, which target on tokens, spans, and span pairs. In particular, a span encoder is trained to recover a random shuffling of tokens in a span, and a span pair encoder is trained to predict positive pairs that are from the same sentences and negative pairs that are from different sentences using contrastive loss. Experimental results show that the proposed pre-training method outperforms distantly supervised pre-training, and achieves promising performance on two entity relation extraction benchmark datasets (ACE05, SciERC).- Anthology ID:
- 2020.emnlp-main.132
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1692–1705
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.132
- DOI:
- 10.18653/v1/2020.emnlp-main.132
- Cite (ACL):
- Yijun Wang, Changzhi Sun, Yuanbin Wu, Junchi Yan, Peng Gao, and Guotong Xie. 2020. Pre-training Entity Relation Encoder with Intra-span and Inter-span Information. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1692–1705, Online. Association for Computational Linguistics.
- Cite (Informal):
- Pre-training Entity Relation Encoder with Intra-span and Inter-span Information (Wang et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2020.emnlp-main.132.pdf