@inproceedings{pagnoni-etal-2025-byte,
    title = "Byte Latent Transformer: Patches Scale Better Than Tokens",
    author = "Pagnoni, Artidoro  and
      Pasunuru, Ramakanth  and
      Rodriguez, Pedro  and
      Nguyen, John  and
      Muller, Benjamin  and
      Li, Margaret  and
      Zhou, Chunting  and
      Yu, Lili  and
      Weston, Jason E  and
      Zettlemoyer, Luke  and
      Ghosh, Gargi  and
      Lewis, Mike  and
      Holtzman, Ari  and
      Iyer, Srini",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.453/",
    pages = "9238--9258",
    ISBN = "979-8-89176-251-0",
    abstract = "We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models {--} up to 8B parameters and 4T training bytes {--} demonstrating the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. For fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size."
}