@inproceedings{meyur-etal-2025-weqa,
    title = "{W}e{QA}: A Benchmark for Retrieval Augmented Generation in Wind Energy Domain",
    author = "Meyur, Rounak  and
      Phan, Hung  and
      Wagle, Sridevi  and
      Strube, Jan  and
      Halappanavar, Mahantesh  and
      Horawalavithana, Sameera  and
      Acharya, Anurag  and
      Munikoti, Sai",
    editor = "Atwell, Katherine  and
      Biester, Laura  and
      Borah, Angana  and
      Dementieva, Daryna  and
      Ignat, Oana  and
      Kotonya, Neema  and
      Liu, Ziyi  and
      Wan, Ruyuan  and
      Wilson, Steven  and
      Zhao, Jieyu",
    booktitle = "Proceedings of the Fourth Workshop on NLP for Positive Impact (NLP4PI)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.nlp4pi-1.20/",
    doi = "10.18653/v1/2025.nlp4pi-1.20",
    pages = "239--251",
    ISBN = "978-1-959429-19-7",
    abstract = "Wind energy project assessments present significant challenges for decision-makers, who must navigate and synthesize hundreds of pages of environmental and scientific documentation. These documents often span different regions and project scales, covering multiple domains of expertise. This process traditionally demands immense time and specialized knowledge from decision-makers. The advent of Large Language Models (LLM) and Retrieval Augmented Generation (RAG) approaches offer a transformative solution, enabling rapid, accurate cross-document information retrieval and synthesis. As the landscape of Natural Language Processing (NLP) and text generation continues to evolve, benchmarking becomes essential to evaluate and compare the performance of different RAG-based LLMs. In this paper, we present a comprehensive framework to generate a domain relevant RAG benchmark. Our framework is based on automatic question-answer generation with Human (domain experts)-AI (LLM) teaming. As a case study, we demonstrate the framework by introducing WeQA, a first-of-its-kind benchmark on the wind energy domain which comprises of multiple scientific documents/reports related to environmental aspects of wind energy projects. Our framework systematically evaluates RAG performance using diverse metrics and multiple question types with varying complexity level, providing a foundation for rigorous assessment of RAG-based systems in complex scientific domains and enabling researchers to identify areas for improvement in domain-specific applications."
}