@inproceedings{park-etal-2025-powerformer,
    title = "Powerformer: Efficient and High-Accuracy Privacy-Preserving Language Model with Homomorphic Encryption",
    author = "Park, Dongjin  and
      Lee, Eunsang  and
      Lee, Joon-Woo",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.543/",
    pages = "11090--11111",
    ISBN = "979-8-89176-251-0",
    abstract = "We propose Powerformer, an efficient homomorphic encryption (HE)-based privacy-preserving language model (PPLM) designed to reduce computation overhead while maintaining model performance. Powerformer incorporates three key techniques to optimize encrypted computations:1. A novel distillation technique that replaces softmax and layer normalization (LN) with computationally efficient power and linear functions, ensuring no performance degradation while enabling seamless encrypted computation.2. A pseudo-sign composite approximation method that accurately approximates GELU and tanh functions with minimal computational overhead.3. A homomorphic matrix multiplication algorithm specifically optimized for Transformer models, enhancing efficiency in encrypted environments.By integrating these techniques, Powerformer based on the BERT-base model achieves a 45{\%} reduction in computation time compared to the state-of-the-art HE-based PPLM without any loss in accuracy."
}