Large language models (LLMs) are widely deployed in real-world applications, yet their safety alignment often fails to generalize beyond the specific linguistic formats seen during training. Prior work has shown that mismatched generalization can lead to alignment failures, but these studies typically rely on fixed or narrow transformation schemes. In this work, we probe safety alignment generalization using language game jailbreaks, a class of linguistically structured transformations that alter surface form while preserving fluency and semantic recoverability. We further introduce custom language games, which parameterize and vary transformation rules, enabling controlled exploration of alignment behavior across closely related linguistic variants. To scale this analysis, we propose AutoLanJail, an automated framework for discovering and refining language game-based jailbreaks. Experiments across open-source and closed-source LLMs show that safety fine-tuning is highly format-specific: defenses trained on one linguistic form fail to generalize to even minimal variations. These findings reveal a structural limitation of current fine-tuning-based alignment methods and highlight the need for safety evaluations that account for systematic linguistic variation.

Although large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, reliable evaluation remains a critical challenge due to data contamination, opaque operation, and subjective preferences. To address these issues, we propose League of LLMs (LOL), a novel benchmark-free evaluation paradigm that organizes multiple LLMs into a self-governed league for multi-round mutual evaluation. LOL integrates four core criteria (dynamic, transparent, objective, and professional) to mitigate key limitations of existing paradigms. Experiments on eight mainstream LLMs in mathematics and programming demonstrate that LOL can effectively distinguish LLM capabilities while maintaining high internal ranking stability (Top-k consistency = 70.7%). Beyond ranking, LOL reveals empirical findings that are difficult for traditional paradigms to capture. For instance, “memorization-based answering” behaviors are observed in some models, and higher in-family scores are found in the OpenAI model family (𝛥 = 9, p < 0.05). Finally, we make our framework and code publicly available as a valuable complement to the current LLM evaluation ecosystem.