@inproceedings{liu-etal-2025-training,
    title = "Training with Fewer Bits: Unlocking Edge {LLM}s Training with Stochastic Rounding",
    author = "Liu, Taowen  and
      Andronic, Marta  and
      Gunduz, Deniz  and
      Constantinides, George Anthony",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.784/",
    doi = "10.18653/v1/2025.findings-emnlp.784",
    pages = "14531--14546",
    ISBN = "979-8-89176-335-7",
    abstract = "LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors{---}especially batch size{---}remains underexplored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during backpropagation. Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights. Our experiments validate these theoretical insights."
}