DET: A Dual-Encoding Transformer for Relational Graph Embedding

Lingbing Guo, Zhuo Chen, Jiaoyan Chen, Qiang Zhang, Huajun Chen

Abstract
Despite recent successes in natural language processing and computer vision, Transformer faces scalability issues when processing graphs, e.g., computing the full node-to-node attention on knowledge graphs (KGs) with million of entities is still infeasible. The existing methods mitigate this problem by considering only the local neighbors, sacrificing the Transformer’s ability to attend to elements at any distance. This paper proposes a new Transformer architecture called Dual-Encoding Transformer (DET). DET comprises a structural encoder to aggregate information from nearby neighbors, and a semantic encoder to seek for semantically relevant nodes. We adopt a semantic neighbor search approach inspired by multiple sequence alignment (MSA) algorithms used in biological sciences. By stacking the two encoders alternately, similar to the MSA Transformer for protein representation, our method achieves superior performance compared to state-of-the-art attention-based methods on complex relational graphs like KGs and citation networks. Additionally, DET remains competitive for smaller graphs such as molecules.
Anthology ID:
2024.lrec-main.419
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
4685–4696
Language:
URL:
https://aclanthology.org/2024.lrec-main.419
DOI:
Bibkey:
Cite (ACL):
Lingbing Guo, Zhuo Chen, Jiaoyan Chen, Qiang Zhang, and Huajun Chen. 2024. DET: A Dual-Encoding Transformer for Relational Graph Embedding. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4685–4696, Torino, Italia. ELRA and ICCL.
Cite (Informal):
DET: A Dual-Encoding Transformer for Relational Graph Embedding (Guo et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/add_acl24_videos/2024.lrec-main.419.pdf
PDF Search