DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding

Zilong Wang, Mingjie Zhan, Xuebo Liu, Ding Liang

Abstract
Form understanding depends on both textual contents and organizational structure. Although modern OCR performs well, it is still challenging to realize general form understanding because forms are commonly used and of various formats. The table detection and handcrafted features in previous works cannot apply to all forms because of their requirements on formats. Therefore, we concentrate on the most elementary components, the key-value pairs, and adopt multimodal methods to extract features. We consider the form structure as a tree-like or graph-like hierarchy of text fragments. The parent-child relation corresponds to the key-value pairs in forms. We utilize the state-of-the-art models and design targeted extraction modules to extract multimodal features from semantic contents, layout information, and visual images. A hybrid fusion method of concatenation and feature shifting is designed to fuse the heterogeneous features and provide an informative joint representation. We adopt an asymmetric algorithm and negative sampling in our model as well. We validate our method on two benchmarks, MedForm and FUNSD, and extensive experiments demonstrate the effectiveness of our method.
Anthology ID:
2020.findings-emnlp.80
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
898–908
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.80
DOI:
10.18653/v1/2020.findings-emnlp.80
Bibkey:
Cite (ACL):
Zilong Wang, Mingjie Zhan, Xuebo Liu, and Ding Liang. 2020. DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 898–908, Online. Association for Computational Linguistics.
Cite (Informal):
DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding (Wang et al., Findings 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-3/2020.findings-emnlp.80.pdf
Data
FUNSD
PDF Search