Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition
Gang Zhao, Guanting Dong, Yidong Shi, Haolong Yan, Weiran Xu, Si Li
Abstract
Multimodal Named Entity Recognition (MNER) faces two specific challenges: 1) How to capture useful entity-related visual information. 2) How to alleviate the interference of visual noise. Previous works have gained progress by improving interacting mechanisms or seeking for better visual features. However, existing methods neglect the integrity of entity semantics and conduct cross-modal interaction at token-level, which cuts apart the semantics of entities and makes non-entity tokens easily interfered with by irrelevant visual noise. Thus in this paper, we propose an end-to-end heterogeneous Graph-based Entity-level Interacting model (GEI) for MNER. GEI first utilizes a span detection subtask to obtain entity representations, which serve as the bridge between two modalities. Then, the heterogeneous graph interacting network interacts entity with object nodes to capture entity-related visual information, and fuses it into only entity-associated tokens to rid non-entity tokens of the visual noise. Experiments on two widely used datasets demonstrate the effectiveness of our method. Our code will be available at https://github.com/GangZhao98/GEI.- Anthology ID:
- 2022.findings-emnlp.473
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6345–6350
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.473
- DOI:
- 10.18653/v1/2022.findings-emnlp.473
- Cite (ACL):
- Gang Zhao, Guanting Dong, Yidong Shi, Haolong Yan, Weiran Xu, and Si Li. 2022. Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6345–6350, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition (Zhao et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2022.findings-emnlp.473.pdf