@inproceedings{liu-etal-2025-cola,
    title = "{C}o{LA}: Compute-Efficient Pre-Training of {LLM}s via Low-Rank Activation",
    author = "Liu, Ziyue  and
      Zhang, Ruijie  and
      Wang, Zhengyang  and
      Yan, Mingsong  and
      Yang, Zi  and
      Hovland, Paul D.  and
      Nicolae, Bogdan  and
      Cappello, Franck  and
      Tang, Sui  and
      Zhang, Zheng",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.emnlp-main.230/",
    doi = "10.18653/v1/2025.emnlp-main.230",
    pages = "4627--4645",
    ISBN = "979-8-89176-332-6",
    abstract = "The full-size MLPs and the projection layers in attention introduce tremendous model sizes of large language models (LLMs), consuming extensive computational resources in pre-training. We empirically observe that the activations of pre-trained LLMs exhibit low-rank property. Motivated by such observations, we propose **CoLA** and its memory-efficient implementation, **CoLA-M**, to replace these full-size layers with compute-efficient **auto-encoders** that naturally enforce low-rank activations throughout training. This fundamental architectural change eliminates the activation redundancy and significantly boosts model capacity and training efficiency. Experiments on LLaMA models with 60 million to 7 billion parameters show that CoLA reduces the computing cost by $\bf 2\pmb{\times}$ and improves training throughput by $\bf 1.86\pmb{\times}$ while maintaining full-rank level performance. CoLA-M further squeezes memory cost without sacrificing throughput, offering a pre-training approach with collectively superior parameter, computing, and memory efficiency. The LLMs produced are also $\bf 2\pmb{\times}$ smaller, enabling faster inference with lower memory cost on resource-constrained platforms."
}