@inproceedings{ghaddar-etal-2025-charpeval,
    title = "{CHARPEVAL}: Benchmarking Large Language Models' Contextual Reasoning in Knowledge-Grounded Dialogue",
    author = "Ghaddar, Abbas  and
      Alfonso-Hermelo, David  and
      Langlais, Philippe  and
      Chen, Boxing  and
      Parthasarathi, Prasanna",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.860/",
    pages = "16764--16775",
    ISBN = "979-8-89176-256-5",
    abstract = "This paper presents CHARPEVAL, a challenging benchmark specifically designed to evaluate the ability of Large Language Models (LLMs) to perform contextualized reasoning in knowledge-grounded dialogue scenarios. The task involves selecting the correct response from 6 options, including 5 manually crafted distractors, given the conversation history. Extensive benchmarking experiments with a diverse set of state-of-the-art open-weight LLMs show poor performance on CHARPEVAL due to their inability to effectively reason over discontinuous chunks of text across the input. Our analysis reveals systematic error patterns across models with different properties, highlighting the need to improve LLMs beyond simply scaling-up data and compute. CHARPEVAL is publicly available at https://huggingface.co/datasets/huawei-noah/CHARP."
}