@inproceedings{zhao-etal-2024-findver,
    title = "{F}in{DV}er: Explainable Claim Verification over Long and Hybrid-content Financial Documents",
    author = "Zhao, Yilun  and
      Long, Yitao  and
      Jiang, Tintin  and
      Wang, Chengye  and
      Chen, Weiyuan  and
      Liu, Hongjun  and
      Tang, Xiangru  and
      Zhang, Yiming  and
      Zhao, Chen  and
      Cohan, Arman",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2024.emnlp-main.818/",
    doi = "10.18653/v1/2024.emnlp-main.818",
    pages = "14739--14752",
    abstract = "We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 4,000 expert-annotated examples across four subsets, each focusing on a type of scenario that frequently arises in real-world financial domains. We assess a broad spectrum of 25 LLMs under long-context and RAG settings. Our results show that even the current best-performing system (i.e., GPT-4o) significantly lags behind human experts. Our detailed findings and insights highlight the strengths and limitations of existing LLMs in this new task. We believe FinDVer can serve as a valuable benchmark for evaluating LLM capabilities in claim verification over complex, expert-domain documents."
}