@inproceedings{biswas-etal-2025-evaluating,
    title = "Evaluating Retrieval Augmented Generation to Communicate {UK} Climate Change Information",
    author = "Biswas, Arjun  and
      Chahout, Hatim  and
      Pigram, Tristan  and
      Dong, Hang  and
      Williams, Hywel T.p.  and
      Fung, Fai  and
      Xie, Hailun",
    editor = "Dutia, Kalyan  and
      Henderson, Peter  and
      Leippold, Markus  and
      Manning, Christoper  and
      Morio, Gaku  and
      Muccione, Veruska  and
      Ni, Jingwei  and
      Schimanski, Tobias  and
      Stammbach, Dominik  and
      Singh, Alok  and
      Su, Alba (Ruiran)  and
      A. Vaghefi, Saeid",
    booktitle = "Proceedings of the 2nd Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2025)",
    month = jul,
    year = "2025",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.climatenlp-1.9/",
    pages = "126--141",
    ISBN = "979-8-89176-259-6",
    abstract = "There is a huge demand for information about climate change across all sectors as societies seek to mitigate and adapt to its impacts. However, the volume and complexity of climate information, which takes many formats including numerical, text, and tabular data, can make good information hard to access. Here we use Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) to create an AI agent that provides accurate and complete information from the United Kingdom Climate Projections 2018 (UKCP18) data archive. To overcome the problematic hallucinations associated with LLMs, four phases of experiments were performed to optimize different components of our RAG framework, combining various recent retrieval strategies. Performance was evaluated using three statistical metrics (faithfulness, relevance, coverage) as well as human evaluation by subject matter experts. Results show that the best model significantly outperforms a generic LLM (GPT-3.5) and has high-quality outputs with positive ratings by human experts. The UKCP Chatbot developed here will enable access at scale to the UKCP18 climate archives, offering an important case study of using RAG-based LLM systems to communicate climate information."
}