Merge and Recognize: A Geometry and 2D Context Aware Graph Model for Named Entity Recognition from Visual Documents
Chuwei Luo, Yongpan Wang, Qi Zheng, Liangchen Li, Feiyu Gao, Shiyu Zhang
Abstract
Named entity recognition (NER) from visual documents, such as invoices, receipts or business cards, is a critical task for visual document understanding. Most classical approaches use a sequence-based model (typically BiLSTM-CRF framework) without considering document structure. Recent work on graph-based model using graph convolutional networks to encode visual and textual features have achieved promising performance on the task. However, few attempts take geometry information of text segments (text in bounding box) in visual documents into account. Meanwhile, existing methods do not consider that related text segments which need to be merged to form a complete entity in many real-world situations. In this paper, we present GraphNEMR, a graph-based model that uses graph convolutional networks to jointly merge text segments and recognize named entities. By incorporating geometry information from visual documents into our model, richer 2D context information is generated to improve document representations. To merge text segments, we introduce a novel mechanism that captures both geometry information as well as semantic information based on pre-trained language model. Experimental results show that the proposed GraphNEMR model outperforms both sequence-based and graph-based SOTA methods significantly.- Anthology ID:
- 2020.textgraphs-1.3
- Volume:
- Proceedings of the Graph-based Methods for Natural Language Processing (TextGraphs)
- Month:
- December
- Year:
- 2020
- Address:
- Barcelona, Spain (Online)
- Venue:
- TextGraphs
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 24–34
- Language:
- URL:
- https://aclanthology.org/2020.textgraphs-1.3
- DOI:
- 10.18653/v1/2020.textgraphs-1.3
- Cite (ACL):
- Chuwei Luo, Yongpan Wang, Qi Zheng, Liangchen Li, Feiyu Gao, and Shiyu Zhang. 2020. Merge and Recognize: A Geometry and 2D Context Aware Graph Model for Named Entity Recognition from Visual Documents. In Proceedings of the Graph-based Methods for Natural Language Processing (TextGraphs), pages 24–34, Barcelona, Spain (Online). Association for Computational Linguistics.
- Cite (Informal):
- Merge and Recognize: A Geometry and 2D Context Aware Graph Model for Named Entity Recognition from Visual Documents (Luo et al., TextGraphs 2020)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2020.textgraphs-1.3.pdf