@inproceedings{yu-etal-2025-ekrag,
    title = "{EKRAG}: Benchmark {RAG} for Enterprise Knowledge Question Answering",
    author = "Yu, Tan  and
      Zhou, Wenfei  and
      Leiyang, Leiyang  and
      Shukla, Aaditya  and
      Mmadugula, Mmadugula  and
      Gundecha, Pritam  and
      Burnett, Nicholas  and
      Xu, Anbang  and
      Viseth, Viseth  and
      Tbar, Tbar  and
      Akkiraju, Rama  and
      Zhang, Vivienne",
    editor = "Shi, Weijia  and
      Yu, Wenhao  and
      Asai, Akari  and
      Jiang, Meng  and
      Durrett, Greg  and
      Hajishirzi, Hannaneh  and
      Zettlemoyer, Luke",
    booktitle = "Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.knowledgenlp-1.13/",
    pages = "152--159",
    ISBN = "979-8-89176-229-9",
    abstract = "Retrieval-augmented generation (RAG) offers a robust solution for developing enterprise internal virtual assistants by leveraging domain-specific knowledge and utilizing information from frequently updated corporate document repositories. In this work, we introduce the Enterprise-Knowledge RAG (EKRAG) dataset to benchmark RAG for enterprise knowledge question-answering (QA) across a diverse range of corporate documents, such as product releases, technical blogs, and financial reports. Using EKRAG, we systematically evaluate various retrieval models and strategies tailored for corporate content. We propose novel embedding-model (EM)-as-judge and ranking-model (RM)-as-judge approaches to assess answer quality in the context of enterprise information. Combining these with the existing LLM-as-judge method, we then comprehensively evaluate the correctness, relevance, and faithfulness of generated answers to corporate queries. Our extensive experiments shed light on optimizing RAG pipelines for enterprise knowledge QA, providing valuable guidance for practitioners. This work contributes to enhancing information retrieval and question-answering capabilities in corporate environments that demand high degrees of factuality and context-awareness."
}