CAPTAIN: Semantic Feature Injection for Memorization Mitigation in Text-to-Image Diffusion Models
Tong Zhang, Carlos Hinojosa, Bernard Ghanem
2025-12-16
Summary
This paper addresses the issue of diffusion models, which are powerful AI image generators, sometimes recreating images they were trained on, which can cause problems with privacy and copyright.
What's the problem?
As diffusion models become more popular and are used to create a lot of images, there's a risk they'll accidentally copy and reproduce specific images from their training data. This is a problem because those training images might contain private information or be protected by copyright, and simply asking the AI to create something new shouldn't result in a direct copy of existing work. Current methods to prevent this often involve tweaking how the AI follows instructions or changing the initial input, but these can sometimes make the generated images less accurate to what you actually asked for.
What's the solution?
The researchers developed a new technique called CAPTAIN that works *after* the image generation process has started, without needing to retrain the AI. CAPTAIN works by adding a special type of noise at the beginning to discourage the AI from immediately trying to recreate memorized images. Then, it figures out *when* and *where* in the image creation process the AI is most likely to copy something. Finally, it subtly replaces those copied parts with similar features from different, non-memorized images, ensuring the final image still matches the original request but doesn't directly reproduce the training data.
Why it matters?
This research is important because it offers a way to make diffusion models safer and more respectful of privacy and copyright. By reducing the chance of accidental image reproduction without sacrificing the quality or accuracy of the generated images, CAPTAIN helps unlock the full potential of these powerful AI tools while mitigating potential legal and ethical concerns.
Abstract
Diffusion models can unintentionally reproduce training examples, raising privacy and copyright concerns as these systems are increasingly deployed at scale. Existing inference-time mitigation methods typically manipulate classifier-free guidance (CFG) or perturb prompt embeddings; however, they often struggle to reduce memorization without compromising alignment with the conditioning prompt. We introduce CAPTAIN, a training-free framework that mitigates memorization by directly modifying latent features during denoising. CAPTAIN first applies frequency-based noise initialization to reduce the tendency to replicate memorized patterns early in the denoising process. It then identifies the optimal denoising timesteps for feature injection and localizes memorized regions. Finally, CAPTAIN injects semantically aligned features from non-memorized reference images into localized latent regions, suppressing memorization while preserving prompt fidelity and visual quality. Our experiments show that CAPTAIN achieves substantial reductions in memorization compared to CFG-based baselines while maintaining strong alignment with the intended prompt.