@inproceedings{filice-etal-2025-generating,
    title = "Generating {Q}{\&}{A} Benchmarks for {RAG} Evaluation in Enterprise Settings",
    author = "Filice, Simone  and
      Horowitz, Guy  and
      Carmel, David  and
      Karnin, Zohar  and
      Lewin-Eytan, Liane  and
      Maarek, Yoelle",
    editor = "Rehm, Georg  and
      Li, Yunyao",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.acl-industry.33/",
    pages = "469--484",
    ISBN = "979-8-89176-288-6",
    abstract = "We introduce DataMorgana, a tool for generating synthetic Q{\&}A benchmarks tailored to RAG applications in enterprise settings. DataMorgana enables customization of the generated benchmark according to the expected diverse traffic of the RAG application. It allows for specifying question types and their associated distribution via a lightweight configuration mechanism. We demonstrate via a series of quantitative and qualitative experiments that DataMorgana surpasses existing tools in terms of lexical, syntactic, and semantic diversity of the generated benchmark while maintaining high quality. We run our experiments over domain-specific and general-knowledge public datasets, as well as two private datasets from governmental RAG applications: one for citizens and the other for government employees. The private datasets have been shared with us by AI71, an AI company, which has integrated DataMorgana into its offerings. In addition, DataMorgana has been offered to about 150 researchers worldwide as part of the SIGIR{'}2025 LiveRAG Challenge held in Spring 2025."
}