@inproceedings{chi-etal-2026-mta,
    title = "{MTA}: Multi-Granular Trajectory Alignment for Large Language Model Distillation",
    author = "Chi, Pham Khanh  and
      Dao, Quoc Phong  and
      Nguyen, Thuat  and
      Van, Linh Ngo  and
      Le, Trung  and
      Nguyen, Thanh Hong",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1507/",
    pages = "32669--32684",
    ISBN = "979-8-89176-390-6",
    abstract = "Knowledge distillation is a key technique for compressing large language models (LLMs), but most existing methods align representations at fixed layers or token-level outputs, ignoring how representations evolve across depth. As a result, the student is only weakly guided to capture the teacher{'}s internal relational structure during distillation, which limits knowledge transfer. To address this limitation, we propose \textbf{Multi-Granular Trajectory Alignment (MTA)}, a framework that aligns teacher and student representations along their layer-wise transformation trajectory. MTA adopts a layer-adaptive strategy: lower layers are aligned at the word level to preserve lexical information, while higher layers operate on phrase-level spans (e.g., noun and verb phrases) to capture compositional semantics. We instantiate this idea through a Dynamic Structural Alignment loss that matches the relative geometry among semantic units within each layer. This design is motivated by empirical findings that Transformer representations become increasingly abstract with depth, and is also consistent with linguistic views in which higher-level meaning emerges through the composition of lower-level lexical units. We further incorporate a Hidden Representation Alignment loss to directly align selected teacher{--}student layers. Experiments show that MTA consistently outperforms state-of-the-art baselines on standard benchmarks, with ablations confirming the contribution of each component."
}