@inproceedings{kim-etal-2024-dockd,
    title = "{D}oc{KD}: Knowledge Distillation from {LLM}s for Open-World Document Understanding Models",
    author = "Kim, Sungnyun  and
      Liao, Haofu  and
      Appalaraju, Srikar  and
      Tang, Peng  and
      Tu, Zhuowen  and
      Satzoda, Ravi Kumar  and
      Manmatha, R.  and
      Mahadevan, Vijay  and
      Soatto, Stefano",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.emnlp-main.185/",
    doi = "10.18653/v1/2024.emnlp-main.185",
    pages = "3167--3193",
    abstract = "Visual document understanding (VDU) is a challenging task that involves understanding documents across various modalities (text and image) and layouts (forms, tables, etc.). This study aims to enhance generalizability of small VDU models by distilling knowledge from LLMs. We identify that directly prompting LLMs often fails to generate informative and useful data. In response, we present a new framework (called DocKD) that enriches the data generation process by integrating external document knowledge. Specifically, we provide an LLM with various document elements like key-value pairs, layouts, and descriptions, to elicit open-ended answers. Our experiments show that DocKD produces high-quality document annotations and surpasses the direct knowledge distillation approach that does not leverage external document knowledge. Moreover, student VDU models trained with solely DocKD-generated data is not only comparable to those trained with human-annotated data on in-domain tasks but also significantly excel them on out-of-domain tasks."
}