@inproceedings{wang-etal-2025-dhp,
    title = "{DHP} Benchmark: Are {LLM}s Good {NLG} Evaluators?",
    author = "Wang, Yicheng  and
      Yuan, Jiayi  and
      Chuang, Yu-Neng  and
      Wang, Zhuoer  and
      Liu, Yingchi  and
      Cusick, Mark  and
      Kulkarni, Param  and
      Ji, Zhengping  and
      Ibrahim, Yasser  and
      Hu, Xia",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.451/",
    pages = "8079--8094",
    ISBN = "979-8-89176-195-7",
    abstract = "Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks; this is often referred to as ``LLM-as-a-judge'' paradigm. However, the capabilities of LLMs in evaluating NLG quality remain underexplored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs. This framework leverages hierarchically perturbed text data and statistical tests to systematically measure the NLG evaluation capabilities of LLMs. We re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM families provides critical insight into their strengths and limitations as NLG evaluators. Our dataset is available at https://huggingface.co/datasets/YCWANGVINCE/DHP{\_}Benchmark."
}