Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation
Adithya V Ganesan, Yash Kumar Lal, August Nilsson, H. Schwartz
Abstract
Very large language models (LLMs) perform extremely well on a spectrum of NLP tasks in a zero-shot setting. However, little is known about their performance on human-level NLP problems which rely on understanding psychological concepts, such as assessing personality traits. In this work, we investigate the zero-shot ability of GPT-3 to estimate the Big 5 personality traits from users’ social media posts. Through a set of systematic experiments, we find that zero-shot GPT-3 performance is somewhat close to an existing pre-trained SotA for broad classification upon injecting knowledge about the trait in the prompts. However, when prompted to provide fine-grained classification, its performance drops to close to a simple most frequent class (MFC) baseline. We further analyze where GPT-3 performs better, as well as worse, than a pretrained lexical model, illustrating systematic errors that suggest ways to improve LLMs on human-level NLP tasks. The code for this project is available on Github.- Anthology ID:
- 2023.wassa-1.34
- Volume:
- Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Jeremy Barnes, Orphée De Clercq, Roman Klinger
- Venue:
- WASSA
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 390–400
- Language:
- URL:
- https://aclanthology.org/2023.wassa-1.34
- DOI:
- 10.18653/v1/2023.wassa-1.34
- Cite (ACL):
- Adithya V Ganesan, Yash Kumar Lal, August Nilsson, and H. Schwartz. 2023. Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 390–400, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation (V Ganesan et al., WASSA 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.wassa-1.34.pdf