@inproceedings{ruoxi-etal-2024-dlue,
    title = "{DLUE}: Benchmarking Document Language Understanding",
    author = "Ruoxi, Xu  and
      Hongyu, Lin  and
      Xinyan, Guan  and
      Yingfei, Sun  and
      Le, Sun",
    editor = "Sun, Maosong  and
      Liang, Jiye  and
      Han, Xianpei  and
      Liu, Zhiyuan  and
      He, Yulan",
    booktitle = "Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)",
    month = jul,
    year = "2024",
    address = "Taiyuan, China",
    publisher = "Chinese Information Processing Society of China",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2024.ccl-1.97/",
    pages = "1257--1269",
    language = "eng",
    abstract = "{\textquotedblleft}Understanding documents is central to many real-world tasks but remains a challenging topic.Unfortunately, there is no well-established consensus on how to comprehensively evaluate docu-ment understanding abilities, which significantly hinders the fair comparison and measuring theprogress of the field. To benchmark document understanding researches, this paper summarizesfour representative abilities, i.e., document classification, document structural analysis, docu-ment information extraction, and document transcription. Under the new evaluation framework,we propose Document Language Understanding Evaluation {--} DLUE, a new task suite whichcovers a wide-range of tasks in various forms, domains and document genres. We also systemat-ically evaluate six well-established transformer models and representative LLMs on DLUE, andfind that due to the lengthy content, complicated underlying structure and dispersed knowledge,document understanding is still far from being solved in complex real-world scenarios.{\textquotedblright}"
}