DreamMix: Decoupling Object Attributes for Enhanced Editability in Customized Image Inpainting
Yicheng Yang, Pengxiang Li, Lu Zhang, Liqian Ma, Ping Hu, Siyu Du, Yunzhi Zhuge, Xu Jia, Huchuan Lu
2024-11-26
Summary
This paper introduces DreamMix, a new model for image editing that allows users to insert objects into images and modify their attributes easily, improving the process of customizing images.
What's the problem?
In image editing, especially when inserting new objects into existing scenes, previous methods often focus too much on keeping the original identity of the objects but struggle with allowing users to easily change the characteristics of these new objects. This can make it hard to achieve both a natural look and the desired edits.
What's the solution?
DreamMix solves this problem by using a diffusion-based generative model that can insert target objects into specified locations in an image while also letting users modify their attributes based on text descriptions. The model includes a framework that balances local object insertion with overall visual coherence. It also introduces mechanisms to separate the attributes of objects from their identity, allowing for more flexible and diverse edits. Extensive testing shows that DreamMix effectively maintains the original look of the scene while allowing for detailed modifications.
Why it matters?
This research is important because it enhances the capabilities of image editing tools, making it easier for users to customize images without losing quality or realism. By improving how objects can be added and edited in images, DreamMix can benefit various applications in graphic design, advertising, and content creation, ultimately leading to more creative possibilities.
Abstract
Subject-driven image inpainting has emerged as a popular task in image editing alongside recent advancements in diffusion models. Previous methods primarily focus on identity preservation but struggle to maintain the editability of inserted objects. In response, this paper introduces DreamMix, a diffusion-based generative model adept at inserting target objects into given scenes at user-specified locations while concurrently enabling arbitrary text-driven modifications to their attributes. In particular, we leverage advanced foundational inpainting models and introduce a disentangled local-global inpainting framework to balance precise local object insertion with effective global visual coherence. Additionally, we propose an Attribute Decoupling Mechanism (ADM) and a Textual Attribute Substitution (TAS) module to improve the diversity and discriminative capability of the text-based attribute guidance, respectively. Extensive experiments demonstrate that DreamMix effectively balances identity preservation and attribute editability across various application scenarios, including object insertion, attribute editing, and small object inpainting. Our code is publicly available at https://github.com/mycfhs/DreamMix.