@inproceedings{peng-etal-2025-harmonizing,
    title = "Harmonizing Diverse Models: A Layer-wise Merging Strategy for Consistent Generation",
    author = "Peng, Xujun  and
      Kumar, Anoop  and
      Wu, Jingyu  and
      Glenn, Parker  and
      Liu, Daben",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.emnlp-industry.64/",
    doi = "10.18653/v1/2025.emnlp-industry.64",
    pages = "954--962",
    ISBN = "979-8-89176-333-3",
    abstract = "Retrieval-Augmented Generation (RAG) systems leverage Large Language Models (LLMs) to generate accurate and reliable responses that are grounded in retrieved context. However, LLMs often generate inconsistent outputs for semantically equivalent inputs, a problem exacerbated by limited consistency-focused data and the limitations of existing fine-tuning methods for improving consistency. We propose a new approach combining systematic synthetic data generation, triplet loss for better embeddings, and a novel layer-wise model merging approach. Using consistency-aware weights derived from intermediate layer activations, our method effectively integrates knowledge from specialized models. Experimental results how that our merged model significantly enhances output consistency, achieving approximately 47.5{\%} improvement in response similarity over the baseline, thus offering a practical solution for increasing the the reliability of an industrial RAG system."
}