@inproceedings{huang-etal-2025-governance,
    title = "Governance in Motion: Co-evolution of Constitutions and {AI} models for Scalable Safety",
    author = "Huang, Chenhao  and
      Shen, Ziyu  and
      Ren, Yicong  and
      Zheng, Huiyuan  and
      Zhang, Jiazheng  and
      Chai, Mingxu  and
      Zhang, Ming  and
      Dou, Shihan  and
      Mo, Fan  and
      Shi, Jie  and
      Gui, Tao  and
      Zhang, Qi  and
      Huang, Xuanjing",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.869/",
    pages = "17198--17221",
    ISBN = "979-8-89176-332-6",
    abstract = "Aligning large language models (LLMs) with human preferences is a central challenge for building reliable AI systems. Most existing alignment approaches rely on static signals, such as predefined principles or offline human annotations to guide model behavior toward a fixed approximation of human preferences. However, LLMs can exhibit distributional drift during training, and static alignment mechanisms lack the capacity to adaptively correct misaligned behaviors as they emerge. To address this limitation, we develop a two-stage framework that enables dynamic and continuous alignment. In the first stage, a constitution is continually revised based on observed model behaviors, and models are trained to comply with these evolving principles. In the second stage, this learned constitution is used to guide reinforcement learning, encouraging the model to align with the updated normative signals. We refer to this framework as COCOA: Co-evolution of Constitutions and AI Models. We show that COCOA enables a 7B model to greatly improve safety{---}raising StrongReject score from 0.741 to 0.935 and Safe-RLHF accuracy from 77.76{\%} to 90.64{\%} without human annotations, reaching performance close to much larger state-of-the-art models."
}