@inproceedings{song-tang-2025-memory,
    title = "Memory-Efficient Backpropagation for Fine-Tuning {LLM}s on Resource-Constrained Mobile Devices",
    author = "Song, Congzheng  and
      Tang, Xinyu",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-industry.52/",
    pages = "766--774",
    ISBN = "979-8-89176-333-3",
    abstract = "Fine-tuning large language models (LLMs) with backpropagation{--}even for a subset of parameters such as LoRA{--}can be much more memory-consuming than inference and is often deemed impractical for resource-constrained mobile devices. Alternative methods, such as zeroth-order optimization (ZO), can greatly reduce the memory footprint but come at the cost of significantly slower model convergence (10{\texttimes} to 100{\texttimes} more steps than backpropagation). We propose a memory-efficient implementation of backpropagation (MeBP) on mobile devices that allows flexible trade-offs between memory usage and compute time, while converging faster and achieving better performance than the ZO baseline. We verify the effectiveness of MeBP on an iPhone 15 Pro Max and show that various LLMs, ranging from 0.5B to 4B parameters, can be fine-tuned using less than 1GB of memory. We release an example of the MeBP implementation at \url{https://github.com/apple/ml-mebp}."
}