@inproceedings{zhao-etal-2026-unleashing-low,
    title = "Unleashing Low-Bit Inference on Ascend {NPU}s: A Comprehensive Evaluation of {H}i{F}loat Formats",
    author = "Zhao, Pengxiang  and
      Zhen, Hui-Ling  and
      Li, Xing  and
      Bao, Han  and
      Lin, Weizhe  and
      Yang, Zhiyuan  and
      Wei, Yu Zi  and
      Wang, Xin  and
      Yuan, Mingxuan  and
      Yu, Xianzhi  and
      Dong, Zhenhua",
    editor = "Li, Yunyao  and
      Rehm, Georg  and
      Tu, Mei",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 6: Industry Track)",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-form-platform/2026.acl-industry.57/",
    pages = "844--864",
    ISBN = "979-8-89176-394-4",
    abstract = "As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs. Through rigorous comparison across weight-activation and KV-cache tasks, we provide three key insights: (1) INT8 suits narrow-range data, while floating-point formats excel with high-variance data; (2) in 4-bit regimes, HiF4{'}s hierarchical scaling prevents the accuracy collapse seen in integer formats; and (3) HiFloat is fully compatible with state-of-the-art post-training quantization frameworks. Overall, HiFloat provides a solution for high-efficiency LLM inference on NPUs."
}