Team NLLG submission for Eval4NLP 2023 Shared Task: Retrieval-Augmented In-Context Learning for NLG Evaluation
Daniil Larionov, Vasiliy Viskov, George Kokush, Alexander Panchenko, Steffen Eger
Abstract
In this paper, we propose a retrieval-augmented in-context learning for natural language generation (NLG) evaluation. This method allows practitioners to utilize large language models (LLMs) for various NLG evaluation tasks without any fine-tuning. We apply our approach to Eval4NLP 2023 Shared Task in translation evaluation and summarization evaluation subtasks. The findings suggest that retrieval-augmented in-context learning is a promising approach for creating LLM-based evaluation metrics for NLG. Further research directions include exploring the performance of various publicly available LLM models and identifying which LLM properties help boost the quality of the metric.- Anthology ID:
- 2023.eval4nlp-1.19
- Volume:
- Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems
- Month:
- November
- Year:
- 2023
- Address:
- Bali, Indonesia
- Editors:
- Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, Andreas Rücklé
- Venues:
- Eval4NLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 228–234
- Language:
- URL:
- https://aclanthology.org/2023.eval4nlp-1.19
- DOI:
- 10.18653/v1/2023.eval4nlp-1.19
- Cite (ACL):
- Daniil Larionov, Vasiliy Viskov, George Kokush, Alexander Panchenko, and Steffen Eger. 2023. Team NLLG submission for Eval4NLP 2023 Shared Task: Retrieval-Augmented In-Context Learning for NLG Evaluation. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 228–234, Bali, Indonesia. Association for Computational Linguistics.
- Cite (Informal):
- Team NLLG submission for Eval4NLP 2023 Shared Task: Retrieval-Augmented In-Context Learning for NLG Evaluation (Larionov et al., Eval4NLP-WS 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.eval4nlp-1.19.pdf