@inproceedings{wu-etal-2025-cprm,
    title = "{CPRM}: A {LLM}-based Continual Pre-training Framework for Relevance Modeling in Commercial Search",
    author = "Wu, Kaixin  and
      Ji, Yixin  and
      Chen, Zeyuan  and
      Wang, Qiang  and
      Wang, Cunxiang  and
      Liu, Hong  and
      Ji, Baijun  and
      Jia, Xu  and
      Liu, Zhongyi  and
      Gu, Jinjie  and
      Zhou, Yuan  and
      Mo, Linjian",
    editor = "Chen, Weizhu  and
      Yang, Yi  and
      Kachuee, Mohammad  and
      Fu, Xue-Yong",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-industry.75/",
    pages = "998--1008",
    ISBN = "979-8-89176-194-0",
    abstract = "Relevance modeling between queries and items stands as a pivotal component in commercial search engines, directly affecting the user experience. Given the remarkable achievements of large language models (LLMs) in various natural language processing (NLP) tasks, LLM-based relevance modeling is gradually being adopted within industrial search systems. Nevertheless, foundational LLMs lack domain-specific knowledge and do not fully exploit the potential of in-context learning. Furthermore, structured item text remains underutilized, and there is a shortage in the supply of corresponding queries and background knowledge. We thereby propose CPRM (Continual Pre-training for Relevance Modeling), a framework designed for the continual pre-training of LLMs to address these issues. Our CPRM framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, a novel approach where LLMs are pre-trained on a sequence of related queries or items, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information (e.g., generating summaries and corresponding queries) to further strengthen LLMs. Results on offline experiments and online A/B testing demonstrate that our model achieves convincing performance compared to strong baselines."
}