EduQuick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media
Abstract
This study explores the capacity of large language models (LLMs) to efficiently generate summaries of informal educational content tailored for platforms like TikTok. It also investigates how both humans and LLMs assess the quality of these summaries, based on a series of experiments, exploring the potential replacement of human evaluation with LLMs. Furthermore, the study delves into how experienced content creators perceive the utility of automatic summaries for TikTok videos. We employ strategic prompt selection techniques to guide LLMs in producing engaging summaries based on the characteristics of viral TikTok content, including hashtags, captivating hooks, storytelling, and user engagement. The study leverages OpenAI’s GPT-4 model to generate TikTok content summaries, aiming to align them with the essential features identified. By employing this model and incorporating human evaluation and expert assessment, this research endeavors to shed light on the intricate dynamics of modern content creation, where AI and human ingenuity converge. Ultimately, it seeks to enhance strategies for disseminating and evaluating educational information effectively in the realm of social media.- Anthology ID:
- 2023.eval4nlp-1.4
- Volume:
- Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems
- Month:
- November
- Year:
- 2023
- Address:
- Bali, Indonesia
- Editors:
- Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, Andreas Rücklé
- Venues:
- Eval4NLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 32–48
- Language:
- URL:
- https://aclanthology.org/2023.eval4nlp-1.4
- DOI:
- 10.18653/v1/2023.eval4nlp-1.4
- Cite (ACL):
- Zahra Kolagar, Sebastian Steindl, and Alessandra Zarcone. 2023. EduQuick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 32–48, Bali, Indonesia. Association for Computational Linguistics.
- Cite (Informal):
- EduQuick: A Dataset Toward Evaluating Summarization of Informal Educational Content for Social Media (Kolagar et al., Eval4NLP-WS 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.eval4nlp-1.4.pdf