RealBehavior: A Framework for Faithfully Characterizing Foundation Models’ Human-like Behavior Mechanisms
Enyu Zhou, Rui Zheng, Zhiheng Xi, Songyang Gao, Xiaoran Fan, Zichu Fei, Jingting Ye, Tao Gui, Qi Zhang, Xuanjing Huang
Abstract
Reports of human-like behaviors in foundation models are growing, with psychological theories providing enduring tools to investigate these behaviors. However, current research tends to directly apply these human-oriented tools without verifying the faithfulness of their outcomes. In this paper, we introduce a framework, RealBehavior, which is designed to characterize the humanoid behaviors of models faithfully. Beyond simply measuring behaviors, our framework assesses the faithfulness of results based on reproducibility, internal and external consistency, and generalizability. Our findings suggest that a simple application of psychological tools cannot faithfully characterize all human-like behaviors. Moreover, we discuss the impacts of aligning models with human and social values, arguing for the necessity of diversifying alignment objectives to prevent the creation of models with restricted characteristics.- Anthology ID:
- 2023.findings-emnlp.688
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 10262–10274
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2023.findings-emnlp.688/
- DOI:
- 10.18653/v1/2023.findings-emnlp.688
- Cite (ACL):
- Enyu Zhou, Rui Zheng, Zhiheng Xi, Songyang Gao, Xiaoran Fan, Zichu Fei, Jingting Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. RealBehavior: A Framework for Faithfully Characterizing Foundation Models’ Human-like Behavior Mechanisms. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10262–10274, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- RealBehavior: A Framework for Faithfully Characterizing Foundation Models’ Human-like Behavior Mechanisms (Zhou et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2023.findings-emnlp.688.pdf