@inproceedings{mouilleron-etal-2026-tables,
    title = "When Tables Go Crazy: Evaluating Multimodal Models on {F}rench Financial Documents",
    author = "Mouilleron, Virginie  and
      Lasnier, Th{\'e}o  and
      Mosolova, Anna  and
      Seddah, Djam{\'e}",
    editor = "El-Haj, Mo  and
      Moreno Sandoval, Antonio  and
      Garcia-Serrano, Ana  and
      Chen, Chung-Chi  and
      Rayson, Paul  and
      Torterolo Orta, Yanco Amor  and
      Martinez, Paloma  and
      Porta, Jordi",
    booktitle = "The 7th Financial Narrative Processing Workshop",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "European Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/cawl-year/2026.fnp-1.2/",
    doi = "10.63317/3bk8fk9sek7b",
    pages = "12--27",
    abstract = "Vision-language models (VLMs) perform well on many document understanding tasks, yet their reliability in specialized, non-English domains remains underexplored. This gap is especially critical in finance, where documents mix dense regulatory text, numerical tables, and visual charts, and where extraction errors can have real-world consequences. We introduce SCRIBE FINANCE, the first multimodal benchmark for evaluating French financial document understanding. The dataset contains 1,204 expert-validated questions spanning text extraction, table comprehension, chart interpretation, and multi-turn conversational reasoning, drawn from real investment prospectuses, KIDs, and PRIIPs. We evaluate six open-weight VLMs (8B{--}124B parameters) using an LLM-as-judge protocol. While models achieve strong performance on text and table tasks (85{--}90{\%} accuracy), they struggle with chart interpretation (34{--}62{\%}). Most notably, multi-turn dialogue reveals a sharp failure mode: early mistakes propagate across turns, driving accuracy down to roughly 50{\%} regardless of model size. These results show that current VLMs are effective for well-defined extraction tasks but remain brittle in interactive, multi-step financial analysis. SCRIBE FINANCE offers a challenging benchmark to measure and drive progress in this high-stakes setting."
}