@inproceedings{benyoash-etal-2025-secque,
    title = "{SECQUE}: A Benchmark for Evaluating Real-World Financial Analysis Capabilities",
    author = "BenYoash, Noga  and
      Brief, Menachem  and
      Ovadia, Oded  and
      Shenderovitz, Gil  and
      Mishaeli, Moshik  and
      Lemberg, Rachel  and
      Sheetrit, Eitam",
    editor = "Dhole, Kaustubh  and
      Clinciu, Miruna",
    booktitle = "Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM{\texttwosuperior})",
    month = jul,
    year = "2025",
    address = "Vienna, Austria and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/transition-to-people-yaml/2025.gem-1.16/",
    pages = "212--230",
    ISBN = "979-8-89176-261-9",
    abstract = "We introduce SECQUE, a comprehensive benchmark for evaluating large language models (LLMs) in financial analysis tasks. SECQUE comprises 565 expert-written questions covering SEC filings analysis across four key categories: comparison analysis, ratio calculation, risk assessment, and financial insight generation. To assess model performance, we develop SECQUE-Judge, an evaluation mechanism leveraging multiple LLM-based judges, which demonstrates strong alignment with human evaluations. Additionally, we provide an extensive analysis of various models' performance on our benchmark. By making SECQUE publicly available (https://huggingface.co/datasets/nogabenyoash/SecQue), we aim to facilitate further research and advancements in financial AI."
}