Three Ways of Using Large Language Models to Evaluate Chat
Ondřej Plátek, Vojtech Hudecek, Patricia Schmidtova, Mateusz Lango, Ondrej Dusek
Abstract
This paper describes the systems submitted by team6 for ChatEval, the DSTC 11 Track 4 competition. We present three different approaches to predicting turn-level qualities of chatbot responses based on large language models (LLMs). We report improvement over the baseline using dynamic few-shot examples from a vector store for the prompts for ChatGPT. We also analyze the performance of the other two approaches and report needed improvements for future work. We developed the three systems over just two weeks, showing the potential of LLMs for this task. An ablation study conducted after the challenge deadline shows that the new Llama 2 models are closing the performance gap between ChatGPT and open-source LLMs. However, we find that the Llama 2 models do not benefit from few-shot examples in the same way as ChatGPT.- Anthology ID:
- 2023.dstc-1.14
- Volume:
- Proceedings of The Eleventh Dialog System Technology Challenge
- Month:
- September
- Year:
- 2023
- Address:
- Prague, Czech Republic
- Editors:
- Yun-Nung Chen, Paul Crook, Michel Galley, Sarik Ghazarian, Chulaka Gunasekara, Raghav Gupta, Behnam Hedayatnia, Satwik Kottur, Seungwhan Moon, Chen Zhang
- Venues:
- DSTC | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 113–122
- Language:
- URL:
- https://aclanthology.org/2023.dstc-1.14
- DOI:
- Cite (ACL):
- Ondřej Plátek, Vojtech Hudecek, Patricia Schmidtova, Mateusz Lango, and Ondrej Dusek. 2023. Three Ways of Using Large Language Models to Evaluate Chat. In Proceedings of The Eleventh Dialog System Technology Challenge, pages 113–122, Prague, Czech Republic. Association for Computational Linguistics.
- Cite (Informal):
- Three Ways of Using Large Language Models to Evaluate Chat (Plátek et al., DSTC-WS 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2023.dstc-1.14.pdf