Do large language models resemble humans in language use?
Zhenguang Cai, Xufeng Duan, David Haslett, Shuqi Wang, Martin Pickering
Abstract
It is unclear whether large language models (LLMs) develop humanlike characteristics in language use. We subjected ChatGPT and Vicuna to 12 pre-registered psycholinguistic experiments ranging from sounds to dialogue. ChatGPT and Vicuna replicated the human pattern of language use in 10 and 7 out of the 12 experiments, respectively. The models associated unfamiliar words with different meanings depending on their forms, continued to access recently encountered meanings of ambiguous words, reused recent sentence structures, attributed causality as a function of verb semantics, and accessed different meanings and retrieved different words depending on an interlocutor’s identity. In addition, ChatGPT, but not Vicuna, nonliterally interpreted implausible sentences that were likely to have been corrupted by noise, drew reasonable inferences, and overlooked semantic fallacies in a sentence. Finally, unlike humans, neither model preferred using shorter words to convey less informative content, nor did they use context to resolve syntactic ambiguities. We discuss how these convergences and divergences may result from the transformer architecture. Overall, these experiments demonstrate that LLMs such as ChatGPT (and Vicuna to a lesser extent) are humanlike in many aspects of human language processing.- Anthology ID:
- 2024.cmcl-1.4
- Volume:
- Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Tatsuki Kuribayashi, Giulia Rambelli, Ece Takmaz, Philipp Wicke, Yohei Oseki
- Venues:
- CMCL | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 37–56
- Language:
- URL:
- https://aclanthology.org/2024.cmcl-1.4
- DOI:
- Cite (ACL):
- Zhenguang Cai, Xufeng Duan, David Haslett, Shuqi Wang, and Martin Pickering. 2024. Do large language models resemble humans in language use?. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 37–56, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Do large language models resemble humans in language use? (Cai et al., CMCL-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.cmcl-1.4.pdf