@inproceedings{lee-etal-2025-towards,
    title = "Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of {LLM}s in Book-Length Contexts",
    author = "Lee, Yuho  and
      Deng, Jiaqi  and
      Kim, Nicole Hee-Yeon  and
      Min, Hyangsuk  and
      Yun, Taewon  and
      Ban, Minjeong  and
      Yul, Kim  and
      Song, Hwanjun",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1241/",
    pages = "24412--24436",
    ISBN = "979-8-89176-332-6",
    abstract = "We introduce HAMLET, a holistic and automated framework for evaluating the long-context comprehension of large language models (LLMs). HAMLET structures key information of source texts into a three-level hierarchy at root-, branch-, and leaf-levels, and employs query-focused summarization to evaluate how well models faithfully recall the key information at each level. To validate the reliability of our fully automated pipeline, we conduct a systematic human study, demonstrating that our automatic evaluation achieves over 90{\%} agreement with expert human judgments, while reducing the evaluation cost by up to 25$\times$. HAMLET reveals that LLMs struggle with fine-grained comprehension, especially at the leaf level, and are sensitive to positional effects like the lost-in-the-middle. Analytical queries pose greater challenges than narrative ones, and consistent performance gaps emerge between open-source and proprietary models, as well as across model scales. Our code and dataset are publicly available at https://github.com/DISL-Lab/HAMLET."
}