@inproceedings{matos-etal-2026-benchmarking,
    title = "Benchmarking Retrieval-Augmented Generation for Scientific Knowledge {QA} in {E}uropean {P}ortuguese",
    author = "Matos, Jose  and
      Silva, Catarina  and
      Goncalo Oliveira, Hugo",
    editor = "Rehm, Georg  and
      Dietze, Stefan  and
      Dessi, Danilo  and
      Maynard, Diana  and
      Schimmler, Sonja",
    booktitle = "Proceedings of Natural Scientific Language Processing ({NSLP}) @ {LREC} 2026",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/test-year-match/2026.nslp-1.3/",
    doi = "10.63317/3muergicuxwk",
    pages = "25--31",
    abstract = "Retrieval-Augmented Generation (RAG) enables grounding of model outputs in external evidence, but its impact on European Portuguese (pt-PT) scientific question answering (QA) remains unclear. We present a controlled evaluation of RAG on pt-PT knowledge QA across different scientific domains using the Portuguese test split of the Global MMLU Lite dataset. As external evidence, we use a Portuguese scientific literature knowledge base containing over 32,000 documents converted to Markdown. We benchmark five instruction-tuned small language models (4-12B) and compare closed-book baselines against 16 RAG configurations that vary by: (i) dense retriever specialization (multilingual vs. Portuguese-specific), (ii) reranking (on/off), and (iii) number of retrieved chunks (k {\ensuremath{\in}} {1, 3, 5, 10}). Results suggest that RAG gains are model-dependent. Some models improve consistently, others are highly sensitive to retrieval choices, and some degrade under retrieval noise, especially at larger values of k. Findings highlight the importance of model-specific retrieval tuning and ensuring that the retriever and reranker languages and domains align when deploying RAG systems for Portuguese natural scientific language processing."
}