@inproceedings{tan-etal-2025-personabench,
    title = "{P}ersona{B}ench: Evaluating {AI} Models on Understanding Personal Information through Accessing (Synthetic) Private User Data",
    author = "Tan, Juntao  and
      Yang, Liangwei  and
      Liu, Zuxin  and
      Liu, Zhiwei  and
      R N, Rithesh  and
      Awalgaonkar, Tulika Manoj  and
      Zhang, Jianguo  and
      Yao, Weiran  and
      Zhu, Ming  and
      Kokane, Shirley  and
      Savarese, Silvio  and
      Wang, Huan  and
      Xiong, Caiming  and
      Heinecke, Shelby",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.49/",
    pages = "878--893",
    ISBN = "979-8-89176-256-5",
    abstract = "Personalization is essential for AI assistants, especially in private AI settings where models are expected to interpret users' personal data (e.g., conversations, app usage) to understand their background, preferences, and social context. However, due to privacy concerns, existing academic research lacks direct access to such data, making benchmarking difficult. To fill this gap, we propose a synthetic data pipeline that generates realistic user profiles and private documents, enabling the creation of PersonaBench{---}a benchmark for evaluating models' ability to understand personal information. Using this benchmark, we assess Retrieval-Augmented Generation (RAG) pipelines on personalized questions and find that current models struggle to accurately extract and answer questions even when provided with the full set of user documents, highlighting the need for improved personalization methods."
}