@inproceedings{kim-etal-2023-chatgpt,
    title = "Can {C}hat{GPT} Understand Causal Language in Science Claims?",
    author = "Kim, Yuheun  and
      Guo, Lu  and
      Yu, Bei  and
      Li, Yingya",
    editor = "Barnes, Jeremy  and
      De Clercq, Orph{\'e}e  and
      Klinger, Roman",
    booktitle = "Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, {\&} Social Media Analysis",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2023.wassa-1.33/",
    doi = "10.18653/v1/2023.wassa-1.33",
    pages = "379--389",
    abstract = "This study evaluated ChatGPT{'}s ability to understand causal language in science papers and news by testing its accuracy in a task of labeling the strength of a claim as causal, conditional causal, correlational, or no relationship. The results show that ChatGPT is still behind the existing fine-tuned BERT models by a large margin. ChatGPT also had difficulty understanding conditional causal claims mitigated by hedges. However, its weakness may be utilized to improve the clarity of human annotation guideline. Chain-of-Thoughts were faithful and helpful for improving prompt performance, but finding the optimal prompt is difficult with inconsistent results and the lack of effective method to establish cause-effect between prompts and outcomes, suggesting caution when generalizing prompt engineering results across tasks or models."
}