@inproceedings{pasch-cha-2025-balancing,
    title = "Balancing Privacy and Utility in Personal {LLM} Writing Tasks: An Automated Pipeline for Evaluating Anonymizations",
    author = "Pasch, Stefan  and
      Cha, Min Chul",
    editor = "Habernal, Ivan  and
      Ghanavati, Sepideh  and
      Jain, Vijayanta  and
      Igamberdiev, Timour  and
      Wilson, Shomir",
    booktitle = "Proceedings of the Sixth Workshop on Privacy in Natural Language Processing",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.privatenlp-main.3/",
    doi = "10.18653/v1/2025.privatenlp-main.3",
    pages = "32--41",
    ISBN = "979-8-89176-246-6",
    abstract = "Large language models (LLMs) are widely used for personalized tasks involving sensitive information, raising privacy concerns. While anonymization techniques exist, their impact on response quality remains underexplored. This paper introduces a fully automated evaluation framework to assess anonymization strategies in LLM-generated responses. We generate synthetic prompts for three personal tasks{---}personal introductions, cover letters, and email writing{---}and apply anonymization techniques that preserve fluency while enabling entity backmapping. We test three anonymization strategies: simple masking, adding context to masked entities, and pseudonymization. Results show minimal response quality loss (roughly 1 point on a 10-point scale) while achieving 97{\%}-99{\%} entity masking. Responses generated with Llama 3.3:70b perform best with simple entity masking, while GPT-4o benefits from contextual cues. This study provides a framework and empirical insights into balancing privacy protection and response quality in LLM applications."
}