@inproceedings{zhou-etal-2025-quzo,
    title = "{Q}u{ZO}: Quantized Zeroth-Order Fine-Tuning for Large Language Models",
    author = "Zhou, Jiajun  and
      Yang, Yifan  and
      Zhen, Kai  and
      Liu, Ziyue  and
      Zhao, Yequan  and
      Banijamali, Ershad  and
      Mouchtaris, Athanasios  and
      Wong, Ngai  and
      Zhang, Zheng",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.271/",
    pages = "5341--5359",
    ISBN = "979-8-89176-332-6",
    abstract = "Large Language Models (LLMs) are often quantized to lower precision to reduce the memory cost and latency in inference. However, quantization often degrades model performance, thus fine-tuning is required for various downstream tasks. Traditional fine-tuning methods such as stochastic gradient descent and Adam optimization require backpropagation, which is error-prone in the low-precision settings. To overcome these limitations, we propose the Quantized Zeroth-Order (QuZO) framework, specifically designed for fine-tuning LLMs through low-precision (e.g., 4- or 8-bit) forward passes. Our method avoids the low-precision straight-through estimator, which requires backward computation, and instead utilizes optimized stochastic rounding to mitigate increased bias. QuZO simplifies the training process, while achieving results comparable to first-order methods in ${\rm FP}8$ and superior accuracy in ${\rm INT}8$ and ${\rm INT}4$ training. Experiments demonstrate that QuZO achieves competitive performance on classification, multi-choice, and generation tasks under low-bit training, including zero-shot reasoning tasks. Notably, QuZO incurs minimal overhead and reduces memory consumption by $2.94 \times${--}$5.47 \times$ compared to quantized first-order methods during LLaMA-7B fine-tuning."
}