@inproceedings{chen-etal-2025-layer,
    title = "Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge",
    author = "Chen, Yan-Lun  and
      Wei, Yi-Ru  and
      Hsu, Chia-Yi  and
      Yu, Chia-Mu  and
      Huang, Chun-Ying  and
      Lin, Ying-Dar  and
      Wu, Yu-Sung  and
      Lee, Wei-Bin",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.644/",
    doi = "10.18653/v1/2025.findings-emnlp.644",
    pages = "12033--12054",
    ISBN = "979-8-89176-335-7",
    abstract = "Large language models (LLMs) demonstrate strong task-specific capabilities through fine-tuning, but merging multiple fine-tuned models often leads to degraded performance due to overlapping instruction-following components. Task Arithmetic (TA), which combines task vectors derived from fine-tuning, enables multi-task learning and task forgetting but struggles to isolate task-specific knowledge from general instruction-following behavior. To address this, we propose Layer-Aware Task Arithmetic (LATA), a novel approach that assigns layer-specific weights to task vectors based on their alignment with instruction-following or task-specific components. By amplifying task-relevant layers and attenuating instruction-following layers, LATA improves task learning and forgetting performance while preserving overall model utility. Experiments on multiple benchmarks, including WikiText-2, GSM8K, and HumanEval, demonstrate that LATA outperforms existing methods in both multi-task learning and selective task forgetting, achieving higher task accuracy and alignment with minimal degradation in output quality. Our findings highlight the importance of layer-wise analysis in disentangling task-specific and general-purpose knowledge, offering a robust framework for efficient model merging and editing."
}