Graph-Induced Transformers for Efficient Multi-Hop Question Answering

Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng

Abstract
A graph is a suitable data structure to represent the structural information of text. Recently, multi-hop question answering (MHQA) tasks, which require inter-paragraph/sentence linkages, have come to exploit such properties of a graph. Previous approaches to MHQA relied on leveraging the graph information along with the pre-trained language model (PLM) encoders. However, this trend exhibits the following drawbacks: (i) sample inefficiency while training in a low-resource setting; (ii) lack of reusability due to changes in the model structure or input. Our work proposes the Graph-Induced Transformer (GIT) that applies graph-derived attention patterns directly into a PLM, without the need to employ external graph modules. GIT can leverage the useful inductive bias of graphs while retaining the unperturbed Transformer structure and parameters. Our experiments on HotpotQA successfully demonstrate both the sample efficient characteristic of GIT and its capacity to replace the graph modules while preserving model performance.
Anthology ID:
2022.emnlp-main.702
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10288–10294
Language:
URL:
https://aclanthology.org/2022.emnlp-main.702
DOI:
10.18653/v1/2022.emnlp-main.702
Bibkey:
Cite (ACL):
Giwon Hong, Jeonghwan Kim, Junmo Kang, and Sung-Hyon Myaeng. 2022. Graph-Induced Transformers for Efficient Multi-Hop Question Answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10288–10294, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Graph-Induced Transformers for Efficient Multi-Hop Question Answering (Hong et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-2023-videos/2022.emnlp-main.702.pdf
PDF Search