CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification
Seungone Kim, Se June Joo, Yul Jang, Hyungjoo Chae, Jinyoung Yeo
Abstract
Chain-of-thought (CoT) prompting enables large language models (LLMs) to solve complex reasoning tasks by generating an explanation before the final prediction. Despite it’s promising ability, a critical downside of CoT prompting is that the performance is greatly affected by the factuality of the generated explanation. To improve the correctness of the explanations, fine-tuning language models with explanation data is needed. However, there exists only a few datasets that can be used for such approaches, and no data collection tool for building them. Thus, we introduce CoTEVer, a tool-kit for annotating the factual correctness of generated explanations and collecting revision data of wrong explanations. Furthermore, we suggest several use cases where the data collected with CoTEVer can be utilized for enhancing the faithfulness of explanations. Our toolkit is publicly available at https://github.com/SeungoneKim/CoTEVer.- Anthology ID:
- 2023.eacl-demo.23
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Danilo Croce, Luca Soldaini
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 195–208
- Language:
- URL:
- https://aclanthology.org/2023.eacl-demo.23
- DOI:
- 10.18653/v1/2023.eacl-demo.23
- Cite (ACL):
- Seungone Kim, Se June Joo, Yul Jang, Hyungjoo Chae, and Jinyoung Yeo. 2023. CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 195–208, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification (Kim et al., EACL 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.eacl-demo.23.pdf