As high-quality data becomes harder to obtain, reward models are increasingly important. Beyond the costly RLHF stage, they are now used at inference time to guide LLM generation and in data selection for post-training. These methods bring efficiency and performance gains, but current reward models often fail to prevent untrustworthy behaviors such as privacy leaks and stereotypes. Re-training reward models to address these issues is expensive, since it requires large-scale human preference data. We propose SelfRW, a lightweight intrinsic reward that needs no extra fine-tuning or auxiliary models. By pruning current LLMs to approximate an “trust” and an “untrust” token distribution, we compute the log-probability difference as an auxiliary reward. When integrated into reward-guided sampling, SelfRW significantly reduces untrustworthy outputs while preserving task performance. It also improves reward-guided data selection, yielding better post-trained models. Experiments with two reward models and four LLMs on privacy, bias, and stereotype benchmarks show that combining SelfRW consistently improves trustworthiness (over 10% in privacy tasks and 20% in bias tasks) with minimal impact on general utility benchmarks.

Machine learning models are increasingly deployed in real-world applications, but even aligned models such as Mistral and LLaVA still exhibit unsafe behaviors inherited from pre-training. Current alignment methods like SFT and RLHF primarily encourage models to generate preferred responses, but do not explicitly remove the unsafe subnetworks that trigger harmful outputs. In this work, we introduce a resource-efficient pruning framework that directly identifies and removes parameters associated with unsafe behaviors while preserving model utility. Our method employs a gradient-free attribution mechanism, requiring only modest GPU resources, and generalizes across architectures and quantized variants. Empirical evaluations on ML models show substantial reductions in unsafe generations and improved robustness against jailbreak attacks, with minimal utility loss. From the perspective of the Lottery Ticket Hypothesis, our results suggest that ML models contain “unsafe tickets” responsible for harmful behaviors, and pruning reveals “safety tickets” that maintain performance while aligning outputs. This provides a lightweight, post-hoc alignment strategy suitable for deployment in resource-constrained settings.