@inproceedings{jiang-etal-2026-open,
    title = {Open Schr{\"o}dinger{'}s Closed Box: Identifying Retrieval Augmented Generation in {API}-Accessible Large Language Model Services},
    author = "Jiang, Yukun  and
      Shen, Xinyue  and
      Backes, Michael  and
      Li, Zheng  and
      Zhang, Yang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.208/",
    pages = "4558--4580",
    ISBN = "979-8-89176-390-6",
    abstract = "Large language models (LLMs) are powerful at question-answering but prone to hallucinations due to limited domain-specific or up-to-date knowledge. Retrieval augmented generation (RAG) mitigates this by adding an external retriever and knowledge database, yet RAG remains vulnerable to targeted attacks that degrade outputs or manipulate opinions. Prior attacks typically assume adversaries know the service is RAG-enhanced and may even know deployment details, an assumption often invalid for real-world commercial LLMs that expose only black-box APIs.This opacity also risks misleading users about system capabilities. This work aims to bridge this gap by proposing $\texttt{RAG-ID}$, a framework for $\underline{\text{ID}}$entifying $\underline{\text{RAG}}$ properties in LLM services.We classify adversaries into three knowledge levels and design six attack methods. Experiments show these attacks reliably detect RAG {---} up to 99.97{\%} accuracy with partial or no optional knowledge, and nearly 100{\%} when the LLM and database are known. After detection, $\texttt{RAG-ID}$ can infer finer RAG properties (e.g., deployed LLM and knowledge database). We consider $\texttt{RAG-ID}$ a reconnaissance tool for attackers, a way to facilitate users' transparent selection of LLM services, and a guide for RAG developers in refining security measures."
}