@inproceedings{li-etal-2025-readoc,
    title = "{READ}oc: A Unified Benchmark for Realistic Document Structured Extraction",
    author = "Li, Zichao  and
      Abulaiti, Aizier  and
      Lu, Yaojie  and
      Chen, Xuanang  and
      Zheng, Jia  and
      Lin, Hongyu  and
      Han, Xianpei  and
      Jiang, Shanshan  and
      Dong, Bin  and
      Sun, Le",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.findings-acl.1128/",
    doi = "10.18653/v1/2025.findings-acl.1128",
    pages = "21889--21905",
    ISBN = "979-8-89176-256-5",
    abstract = "Document Structured Extraction (DSE) aims to extract structured content from raw documents. Despite the emergence of numerous DSE systems, their unified evaluation remains inadequate, significantly hindering the field{'}s advancement. This problem is largely attributed to existing benchmark paradigms, which exhibit fragmented and localized characteristics. To offer a thorough evaluation of DSE systems, we introduce a novel benchmark named READoc, which defines DSE as a realistic task of converting unstructured PDFs into semantically rich Markdown. The READoc dataset is derived from 3,576 diverse and real-world documents from arXiv, GitHub, and Zenodo. In addition, we develop a DSE Evaluation S$^3$uite comprising Standardization, Segmentation and Scoring modules, to conduct a unified evaluation of state-of-the-art DSE approaches. By evaluating a range of pipeline tools, expert visual models, and general Vision-Language Models, we identify the gap between current work and the unified, realistic DSE objective for the first time. We aspire that READoc will catalyze future research in DSE, fostering more comprehensive and practical solutions."
}