@inproceedings{dialameh-etal-2025-echo,
    title = "{ECHO}-{LL}a{MA}: Efficient Caching for High-Performance {LL}a{MA} Training",
    author = "Dialameh, Maryam  and
      Karim, Rezaul  and
      Rajabzadeh, Hossein  and
      Mohamed Awad, Omar  and
      Chen, Boxing  and
      Kwon, Hyock Ju  and
      Ahmed, Walid  and
      Liu, Yang",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.emnlp-industry.156/",
    doi = "10.18653/v1/2025.emnlp-industry.156",
    pages = "2252--2269",
    ISBN = "979-8-89176-333-3",
    abstract = "This paper introduces ECHO-LLaMA, an efficient LLaMA architecture designed to improve both the training speed and inference throughput of LLaMA architectures while maintaining its learning capacity. ECHO-LLaMA transforms LLaMA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. Experimental results demonstrate that ECHO-LLaMA achieves up to 77{\%} higher token-per-second throughput during training, up to 16{\%} higher Model FLOPs Utilization (MFU), and up to 14{\%} lower loss when trained on an equal number of tokens. Furthermore, on the 1.1B model, ECHO-LLaMA delivers approximately 7{\%} higher test-time throughput compared to the baseline. By introducing a computationally efficient adaptation mechanism, ECHO-LLaMA offers a scalable and cost-effective solution for pretraining and finetuning large language models, enabling faster and more resource-efficient training without compromising performance."
}