@inproceedings{singh-etal-2026-cross,
    title = "Cross-Tokenizer {LLM} Distillation through a Byte-Level Interface",
    author = "Singh, Avyav Kumar  and
      Wu, Yen-Chen  and
      Cioba, Alexandru  and
      Bernacchia, Alberto  and
      Buffelli, Davide",
    editor = "Mysore, Sheshera  and
      Kumar, Sachin  and
      Balachandran, Vidhisha  and
      Hayati, Shirley Anugrah  and
      Brahman, Faeze  and
      Moussa, Hanane Nour  and
      Salemi, Alireza",
    booktitle = "Proceedings of the Second Workshop on Customizable {NLP}: Progress and Challenges in Customizing {NLP} for a Domain, Application, Group, or Individual ({C}ustom{NLP}4{U})",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.customnlp4u-1.9/",
    pages = "84--96",
    ISBN = "979-8-89176-396-8",
    abstract = "Cross-tokenizer distillation (CTD), the transfer of knowledge from a teacher to a student language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher{'}s output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with{--}and on several benchmarks surpasses{--}significantly more sophisticated CTD methods, across a range of distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and benchmarks remain elusive, underscoring that CTD is still an open problem."
}