@inproceedings{zhou-etal-2026-balancing,
    title = "Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies",
    author = "Zhou, Changhai  and
      Zhang, Shiyang  and
      Zhou, Yuhua  and
      Gao, Jun  and
      Qiao, Qian  and
      Weng, Shichao  and
      Zhang, Weizhong  and
      Jin, Cheng",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.779/",
    pages = "15885--15896",
    ISBN = "979-8-89176-395-1",
    abstract = "Deploying and fine-tuning Large Language Models (LLMs) on resource-constrained edge devices requires navigating a strict trade-off between memory footprint and task performance. Existing quantization-aware fine-tuning methods typically decouple weight precision and adapter capacity, overlooking that a layer{'}s ability to adapt is constrained by the information preserved in its frozen weights. Layers that are highly sensitive to quantization{---}whether due to representational specialization or accumulated error propagation{---}can become bottlenecks that adapter rank alone cannot recover. To address this issue, we introduce \textbf{QR-Adaptor}, a unified framework that jointly optimizes per-layer quantization bit-width and LoRA rank. We formulate resource allocation as a multi-objective discrete search guided by empirical layer-wise sensitivity, and implement it with a three-stage pipeline comprising KL-based sensitivity profiling, evolutionary exploration, and Bayesian refinement. Extensive experiments across LLaMA and Qwen models, including modern instruction tuning on OpenOrca and comparisons with strong PEFT baselines such as QDoRA, show that QR-Adaptor establishes a strong Pareto frontier: under a strict 4-bit memory budget, it matches or approaches 16-bit baselines while using substantially less memory."
}