@inproceedings{zeng-etal-2025-towards-context,
    title = "Towards Context-Robust {LLM}s: A Gated Representation Fine-tuning Approach",
    author = "Zeng, Shenglai  and
      He, Pengfei  and
      Guo, Kai  and
      Zheng, Tianqi  and
      Lu, Hanqing  and
      Xing, Yue  and
      Liu, Hui",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.506/",
    pages = "10262--10276",
    ISBN = "979-8-89176-251-0",
    abstract = "Large Language Models (LLMs) enhanced with external contexts, such as through retrieval-augmented generation (RAG), often face challenges in handling imperfect evidence. They tend to over-rely on external knowledge, making them vulnerable to misleading and unhelpful contexts. To address this, we propose the concept of context-robust LLMs, which can effectively balance internal knowledge with external context, similar to human cognitive processes. Specifically, context-robust LLMs should rely on external context only when lacking internal knowledge, identify contradictions between internal and external knowledge, and disregard unhelpful contexts. To achieve this goal, we introduce Grft, a lightweight and plug-and-play gated representation fine-tuning approach. Grft consists of two key components: a gating mechanism to detect and filter problematic inputs, and low-rank representation adapters to adjust hidden representations. By training a lightweight intervention function with only 0.0004{\%} of model size on fewer than 200 examples, Grft can effectively adapt LLMs towards context-robust behaviors."
}