Abstract
Instruction-tuned large language models have revolutionized natural language processing and have shown great potential in applications such as conversational agents. These models, such as GPT-4, can not only master language but also solve complex tasks in areas like mathematics, coding, medicine, and law. However, there is still a lack of comprehensive understanding regarding their full potential, primarily due to the black-box nature of many models and lack of holistic evaluation. To address these challenges, we present InstructEval, a more comprehensive evaluation suite designed specifically for instruction-tuned large language models. Unlike previous works, our evaluation involves a rigorous assessment of models based on problem-solving, writing ability, and alignment to human values. We take a holistic approach to analyze various factors affecting model performance, including the pretraining foundation, instruction-tuning data, and training methods. Our findings reveal that the quality of instruction data is a crucial factor in scaling model performance. While open-source models demonstrate impressive writing abilities, there is substantial room for improvement in problem-solving and alignment.- Anthology ID:
- 2024.scalellm-1.4
- Volume:
- Proceedings of the First edition of the Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM 2024)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Antonio Valerio Miceli-Barone, Fazl Barez, Shay Cohen, Elena Voita, Ulrich Germann, Michal Lukasik
- Venues:
- SCALE-LLM | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 35–64
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2024.scalellm-1.4/
- DOI:
- Cite (ACL):
- Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. 2024. InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models. In Proceedings of the First edition of the Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM 2024), pages 35–64, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models (Chia et al., SCALE-LLM 2024)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2024.scalellm-1.4.pdf