Through the Lens of Core Competency: Survey on Evaluation of Large Language Models
Zhuang Ziyu, Chen Qiguang, Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, Ting Liu
Abstract
“From pre-trained language model (PLM) to large language model (LLM), the field of naturallanguage processing (NLP) has witnessed steep performance gains and wide practical uses. Theevaluation of a research field guides its direction of improvement. However, LLMs are extremelyhard to thoroughly evaluate for two reasons. First of all, traditional NLP tasks become inade-quate due to the excellent performance of LLM. Secondly, existing evaluation tasks are difficultto keep up with the wide range of applications in real-world scenarios. To tackle these problems,existing works proposed various benchmarks to better evaluate LLMs. To clarify the numerousevaluation tasks in both academia and industry, we investigate multiple papers concerning LLMevaluations. We summarize 4 core competencies of LLM, including reasoning, knowledge, relia-bility, and safety. For every competency, we introduce its definition, corresponding benchmarks,and metrics. Under this competency architecture, similar tasks are combined to reflect corre-sponding ability, while new tasks can also be easily added into the system. Finally, we give oursuggestions on the future direction of LLM’s evaluation.”- Anthology ID:
- 2023.ccl-2.8
- Volume:
- Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum)
- Month:
- August
- Year:
- 2023
- Address:
- Harbin, China
- Editor:
- Jiajun Zhang
- Venue:
- CCL
- SIG:
- Publisher:
- Chinese Information Processing Society of China
- Note:
- Pages:
- 88–109
- Language:
- English
- URL:
- https://aclanthology.org/2023.ccl-2.8
- DOI:
- Cite (ACL):
- Zhuang Ziyu, Chen Qiguang, Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, and Ting Liu. 2023. Through the Lens of Core Competency: Survey on Evaluation of Large Language Models. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), pages 88–109, Harbin, China. Chinese Information Processing Society of China.
- Cite (Informal):
- Through the Lens of Core Competency: Survey on Evaluation of Large Language Models (Ziyu et al., CCL 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.ccl-2.8.pdf