DocumentNet: Bridging the Data Gap in Document Pre-training
Lijun Yu, Jin Miao, Xiaoyu Sun, Jiayi Chen, Alexander Hauptmann, Hanjun Dai, Wei Wei
Abstract
Document understanding tasks, in particular, Visually-rich Document Entity Retrieval (VDER), have gained significant attention in recent years thanks to their broad applications in enterprise AI. However, publicly available data have been scarce for these tasks due to strict privacy constraints and high annotation costs. To make things worse, the non-overlapping entity spaces from different datasets hinder the knowledge transfer between document types. In this paper, we propose a method to collect massive-scale and weakly labeled data from the web to benefit the training of VDER models. The collected dataset, named DocumentNet, does not depend on specific document types or entity sets, making it universally applicable to all VDER tasks. The current DocumentNet consists of 30M documents spanning nearly 400 document types organized in a four-level ontology. Experiments on a set of broadly adopted VDER tasks show significant improvements when DocumentNet is incorporated into the pre-training for both classic and few-shot learning settings. With the recent emergence of large language models (LLMs), DocumentNet provides a large data source to extend their multimodal capabilities for VDER.- Anthology ID:
- 2023.emnlp-industry.66
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Mingxuan Wang, Imed Zitouni
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 707–722
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-industry.66
- DOI:
- 10.18653/v1/2023.emnlp-industry.66
- Cite (ACL):
- Lijun Yu, Jin Miao, Xiaoyu Sun, Jiayi Chen, Alexander Hauptmann, Hanjun Dai, and Wei Wei. 2023. DocumentNet: Bridging the Data Gap in Document Pre-training. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 707–722, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- DocumentNet: Bridging the Data Gap in Document Pre-training (Yu et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/naacl-24-ws-corrections/2023.emnlp-industry.66.pdf