TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, Bo Zhang
2025-08-20
Summary
This paper discusses a new way to train AI models that create images from text, making them better at following what people like, by improving how they learn from feedback over time.
What's the problem?
Current AI models that generate images from text are good, but they struggle when trying to learn from human preferences using a method called reinforcement learning. This is because they treat every step of image creation the same when assigning credit for success, even though some steps are more important than others, leading to slow learning and not-so-great results.
What's the solution?
The researchers developed a new system called TempFlow-GRPO, which specifically considers the timing of decisions in the image generation process. It uses a technique that creates 'branches' in the AI's learning path to pinpoint which decisions lead to good outcomes and another method that focuses learning on the most impactful stages of image creation, like the early steps, to optimize the process more effectively.
Why it matters?
This advancement is important because it allows AI image generators to become much better at creating images that people actually want, leading to more personalized and useful AI art and design tools.
Abstract
Recent flow matching models for text-to-image generation have achieved remarkable quality, yet their integration with reinforcement learning for human preference alignment remains suboptimal, hindering fine-grained reward-based optimization. We observe that the key impediment to effective GRPO training of flow models is the temporal uniformity assumption in existing approaches: sparse terminal rewards with uniform credit assignment fail to capture the varying criticality of decisions across generation timesteps, resulting in inefficient exploration and suboptimal convergence. To remedy this shortcoming, we introduce TempFlow-GRPO (Temporal Flow GRPO), a principled GRPO framework that captures and exploits the temporal structure inherent in flow-based generation. TempFlow-GRPO introduces two key innovations: (i) a trajectory branching mechanism that provides process rewards by concentrating stochasticity at designated branching points, enabling precise credit assignment without requiring specialized intermediate reward models; and (ii) a noise-aware weighting scheme that modulates policy optimization according to the intrinsic exploration potential of each timestep, prioritizing learning during high-impact early stages while ensuring stable refinement in later phases. These innovations endow the model with temporally-aware optimization that respects the underlying generative dynamics, leading to state-of-the-art performance in human preference alignment and standard text-to-image benchmarks.