SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization
Xuankun Rong, Wenke Huang, Tingfeng Wang, Daiguo Zhou, Bo Du, Mang Ye
2025-11-18
Summary
This paper focuses on making large AI models that can understand both text and images safer to use. These models are good at following instructions and reasoning, but combining text and images can sometimes lead to unexpected and unsafe outputs, even if the individual text or image seems harmless.
What's the problem?
Current AI models that handle both text and images aren't very good at consistently identifying and avoiding unsafe situations. Existing methods to improve safety either rely on humans to guide the AI, which isn't scalable, or they allow the AI to learn on its own but don't provide a clear way to check *why* the AI thinks something is safe or unsafe. Essentially, we need a way to make sure the AI isn't just *appearing* safer, but is actually reasoning about safety in a reliable way.
What's the solution?
The researchers developed a new framework called SafeGRPO. This system builds on an existing method called GRPO, which lets the AI improve itself without human help. SafeGRPO adds a layer of 'rules' to the reward system, so the AI gets rewarded for following specific safety guidelines during its reasoning process. They also created a new dataset called SafeTag-VL-3K, which includes images and text labeled with safety information, to help the AI learn these rules. This encourages the AI to think step-by-step about safety, making its reasoning more structured and easier to understand.
Why it matters?
This work is important because it addresses a critical issue with increasingly powerful AI models: ensuring they are safe and reliable. By making the AI's reasoning process more transparent and verifiable, SafeGRPO helps build trust in these models and reduces the risk of them generating harmful or inappropriate content. It also shows a way to improve safety without constantly needing human intervention, which is crucial for deploying these models in real-world applications.
Abstract
Multimodal large language models (MLLMs) have demonstrated impressive reasoning and instruction-following capabilities, yet their expanded modality space introduces new compositional safety risks that emerge from complex text-image interactions. Such cross-modal couplings can produce unsafe semantics even when individual inputs are benign, exposing the fragile safety awareness of current MLLMs. While recent works enhance safety by guiding models to reason about potential risks, unregulated reasoning traces may compromise alignment; although Group Relative Policy Optimization (GRPO) offers self-rewarded refinement without human supervision, it lacks verifiable signals for reasoning safety. To address this, we propose SafeGRPO a self-rewarded multimodal safety alignment framework that integrates rule-governed reward construction into GRPO, enabling interpretable and verifiable optimization of reasoning safety. Built upon the constructed SafeTag-VL-3K dataset with explicit visual, textual, and combined safety tags, SafeGRPO performs step-guided safety thinking to enforce structured reasoning and behavior alignment, substantially improving multimodal safety awareness, compositional robustness, and reasoning stability across diverse benchmarks without sacrificing general capabilities.