@inproceedings{banar-etal-2026-blleqa,
    title = "b{LL}e{QA}: Benchmarking {LLM}s for Grounded Legal Question-Answering in {F}rench and {D}utch",
    author = "Banar, Nikolay  and
      Lotfi, Ehsan  and
      Van Nooten, Jens  and
      Kliocaite, Marija  and
      Daelemans, Walter",
    editor = "Chen, Canyu  and
      Zhang, Yuji  and
      Li, Zoey Sha  and
      Wang, Zihan  and
      Wang, Qineng  and
      Su, Jinyan  and
      Kargupta, Priyanka  and
      Marjanovi{\'c}, Sara Vera  and
      Pan, Jeff Z.  and
      Bansal, Mohit  and
      Augenstein, Isabelle  and
      Han, Jiawei  and
      Ji, Heng  and
      Li, Manling",
    booktitle = "Proceedings of the 4th Workshop on Towards Knowledgeable Foundation Models ({K}now{FM} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.knowfm-1.4/",
    pages = "34--59",
    ISBN = "979-8-89176-403-3",
    abstract = "Retrieval-augmented generation (RAG) systems can play an important role in making law more accessible. However, large and reliable resources for training and benchmarking such systems remain scarce, especially for under-resourced languages like Dutch. To address this gap, and building on previous work (Louis et al., 2024), we introduce bLLeQA, a bilingual parallel question-answering dataset grounded in Belgian legal resources, both in French and Dutch. The dataset contains aligned questions, answers, and supporting articles in both languages, enabling evaluation of both retrieval and end-to-end RAG pipelines. Using bLLeQA, we benchmark the full RAG pipeline in a zero-shot setting, covering retrieval, citation extraction, refusal behavior, and generation quality. Our experiments show that open-weight models are competitive with proprietary models in retrieval and citation extraction, but lag behind in generation quality in the RAG pipeline. Across all models, refusal capability remains weak, meaning that models do not reliably detect when the provided supporting sources are incomplete. In addition, the end-to-end RAG setup still yields a substantial share of flawed responses, reaching 20{\%} even in the best-case scenario."
}