@inproceedings{zhao-etal-2025-helene,
    title = "{HELENE}: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning {LLM} with Zeroth-order Optimization",
    author = "Zhao, Huaqin  and
      Li, Jiaxi  and
      Pan, Yi  and
      Liang, Shizhe  and
      Yang, Xiaofeng  and
      Dou, Fei  and
      Liu, Tianming  and
      Lu, Jin",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1323/",
    pages = "26055--26078",
    ISBN = "979-8-89176-332-6",
    abstract = "Fine-tuning large language models (LLMs) faces significant memory challenges due to the high cost of back-propagation. MeZO addresses this using zeroth-order (ZO) optimization, matching memory usage to inference but suffering from slow convergence due to varying curvatures across model parameters. To overcome this limitation, We propose HELENE, a scalable and memory-efficient optimizer that integrates annealed A-GNB gradients with diagonal Hessian estimation and layer-wise clipping as a second-order pre-conditioner. HELENE provably accelerates and stabilizes convergence by reducing dependence on total parameter space and scaling with the largest layer dimension. Experiments on RoBERTa-large and OPT-1.3B show up to a 20{\texttimes} speedup over MeZO with an average accuracy improvement of 1.5{\%}. HELENE supports full and parameter-efficient fine-tuning, outperforming several state-of-the-art optimizers."
}