@inproceedings{guo-etal-2025-fineval,
    title = "{F}in{E}val: A {C}hinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models",
    author = "Guo, Xin  and
      Xia, Haotian  and
      Liu, Zhaowei  and
      Cao, Hanyang  and
      Yang, Zhi  and
      Liu, Zhiqiang  and
      Wang, Sizhe  and
      Niu, Jinyi  and
      Wang, Chuqi  and
      Wang, Yanhui  and
      Liang, Xiaolong  and
      Huang, Xiaoming  and
      Zhu, Bing  and
      Wei, Zhongyu  and
      Chen, Yun  and
      Shen, Weining  and
      Zhang, Liwen",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.318/",
    pages = "6258--6292",
    ISBN = "979-8-89176-189-6",
    abstract = "Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored, and their performance on complex tasks like financial agent remains unknown. This paper presents FinEval, a benchmark designed to evaluate LLMs' financial domain knowledge and practical abilities. The dataset contains 8,351 questions categorized into four different key areas: Financial Academic Knowledge, Financial Industry Knowledge, Financial Security Knowledge, and Financial Agent. Financial Academic Knowledge comprises 4,661 multiple-choice questions spanning 34 subjects such as finance and economics. Financial Industry Knowledge contains 1,434 questions covering practical scenarios like investment research. Financial Security Knowledge assesses models through 1,640 questions on topics like application security and cryptography. Financial Agent evaluates tool usage and complex reasoning with 616 questions. FinEval has multiple evaluation settings, including zero-shot, five-shot with chain-of-thought, and assesses model performance using objective and subjective criteria. Our results show that Claude 3.5-Sonnet achieves the highest weighted average score of 72.9 across all financial domain categories under zero-shot setting. Our work provides a comprehensive benchmark closely aligned with Chinese financial domain. The data and the code are available at https://github.com/SUFE-AIFLMLab/FinEval."
}