@inproceedings{li-etal-2025-papillon,
    title = "{PAPILLON}: Privacy Preservation from {I}nternet-based and Local Language Model Ensembles",
    author = "Li, Siyan  and
      Raghuram, Vethavikashini Chithrra  and
      Khattab, Omar  and
      Hirschberg, Julia  and
      Yu, Zhou",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.naacl-long.173/",
    pages = "3371--3390",
    ISBN = "979-8-89176-189-6",
    abstract = "Users can divulge sensitive information to proprietary LLM providers, raising significant privacy concerns. While open-source models, hosted locally on the user`s machine, alleviate some concerns, models that users can host locally are often less capable than proprietary frontier models. Toward preserving user privacy while retaining the best quality, we propose Privacy-Conscious Delegation, a novel task for chaining API-based and local models. We utilize recent public collections of user-LLM interactions to construct a natural benchmark called PUPA, which contains personally identifiable information (PII). To study potential approaches, we devise PAPILLON, a multi-stage LLM pipeline that uses prompt optimization to address a simpler version of our task. Our best pipeline maintains high response quality for 85.5{\%} of user queries while restricting privacy leakage to only 7.5{\%}. We still leave a large margin to the generation quality of proprietary LLMs for future work."
}