@inproceedings{joe-etal-2024-assessing,
    title = "Assessing the Effectiveness of {GPT}-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights",
    author = "Joe, Elphin  and
      Koneru, Sai  and
      Kirchhoff, Christine",
    editor = "Stammbach, Dominik  and
      Ni, Jingwei  and
      Schimanski, Tobias  and
      Dutia, Kalyan  and
      Singh, Alok  and
      Bingler, Julia  and
      Christiaen, Christophe  and
      Kushwaha, Neetu  and
      Muccione, Veruska  and
      A. Vaghefi, Saeid  and
      Leippold, Markus",
    booktitle = "Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2024.climatenlp-1.20/",
    doi = "10.18653/v1/2024.climatenlp-1.20",
    pages = "251--257",
    abstract = "In this research short, we examine the potential of using GPT-4o, a state-of-the-art large language model (LLM) to undertake evidence synthesis and systematic assessment tasks. Traditional workflows for such tasks involve large groups of domain experts who manually review and synthesize vast amounts of literature. The exponential growth of scientific literature and recent advances in LLMs provide an opportunity to complementing these traditional workflows with new age tools. We assess the efficacy of GPT-4o to do these tasks on a sample from the dataset created by the Global Adaptation Mapping Initiative (GAMI) where we check the accuracy of climate change adaptation related feature extraction from the scientific literature across three levels of expertise. Our results indicate that while GPT-4o can achieve high accuracy in low-expertise tasks like geographic location identification, their performance in intermediate and high-expertise tasks, such as stakeholder identification and assessment of depth of the adaptation response, is less reliable. The findings motivate the need for designing assessment workflows that utilize the strengths of models like GPT-4o while also providing refinements to improve their performance on these tasks."
}