@inproceedings{lee-etal-2026-tagquant,
    title = "{TAGQ}uant: Token-Aware Clustering for Group-Wise Quantization",
    author = "Lee, Jaeseong  and
      Hwang, Seung-won  and
      Qiao, Aurick  and
      Yao, Zhewei  and
      He, Yuxiong",
    editor = {Matusevych, Yevgen  and
      Eryi{\u{g}}it, G{\"u}l{\c{s}}en  and
      Aletras, Nikolaos},
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 5: Industry Track)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-industry.18/",
    pages = "253--262",
    ISBN = "979-8-89176-384-5",
    abstract = "Grouping, e.g., grouping channels, which is widely used in current integer-based quantization, has become essential for the emerging MXFP4 format. Ideally, each group should contain channels with similar quantization scales. To guide such groups, existing work clusters the channels using scalar proxy, ignoring the token dimension, which we find suboptimal. In this paper, we propose TAGQuant, a simple yet powerful enhancement for such ``group-wise'' quantization. By strategically shuffling channels to group those with similar token-wise activation distributions, TAGQuant ensures better clustering of large- and small-range values. This shuffle operation is hardware-efficient, and seamlessly integrated into the quantization process with only 0.01x latency overhead. TAGQuant reduces relative GSM8K error in both INT4 and MXFP4 formats, by up to 86{\%} in Llama-3.1-8B-Instruct compared to baselines, validating the effectiveness of our channel shuffling approach for group-wise quantization. Code is publicly available."
}