@inproceedings{lee-etal-2025-llms,
    title = "Do {LLM}s Have Distinct and Consistent Personality? {TRAIT}: Personality Testset designed for {LLM}s with Psychometrics",
    author = "Lee, Seungbeen  and
      Lim, Seungwon  and
      Han, Seungju  and
      Oh, Giyeong  and
      Chae, Hyungjoo  and
      Chung, Jiwan  and
      Kim, Minju  and
      Kwak, Beong-woo  and
      Lee, Yeonsoo  and
      Lee, Dongha  and
      Yeo, Jinyoung  and
      Yu, Youngjae",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.469/",
    pages = "8397--8437",
    ISBN = "979-8-89176-195-7",
    abstract = "Recent advancements in Large Language Models (LLMs) have led to their adaptation in various domains as conversational agents. We wonder: can personality tests be applied to these agents to analyze their behavior, similar to humans? We introduce TRAIT, a new benchmark consisting of 8K multi-choice questions designed to assess the personality of LLMs. TRAIT is built on two psychometrically validated small human questionnaires, Big Five Inventory (BFI) and Short Dark Triad (SD-3), enhanced with the ATOMIC-10X knowledge graph to a variety of real-world scenarios. TRAIT also outperforms existing personality tests for LLMs in terms of reliability and validity, achieving the highest scores across four key metrics: Content Validity, Internal Validity, Refusal Rate, and Reliability. Using TRAIT, we reveal two notable insights into personalities of LLMs: 1) LLMs exhibit distinct and consistent personality, which is highly influenced by their training data (e.g., data used for alignment tuning), and 2) current prompting techniques have limited effectiveness in eliciting certain traits, such as high psychopathy or low conscientiousness, suggesting the need for further research in this direction."
}