RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers
Ke Cao, Jing Wang, Ao Ma, Jiasong Feng, Zhanjie Zhang, Xuanhua He, Shanyuan Liu, Bo Cheng, Dawei Leng, Yuhui Yin, Jie Zhang
2025-02-21
Summary
This paper talks about RelaCtrl, a new way to make AI-generated images and videos more efficient and controllable. It's like teaching an artist to focus on the most important parts of a painting without wasting time and effort on less important details.
What's the problem?
Current methods for controlling how AI generates images and videos use a lot of computer power and memory. They also don't work very efficiently because they treat all parts of the process as equally important, which isn't always true. It's like having an artist spend the same amount of time on every tiny detail of a painting, even the parts that no one will really notice.
What's the solution?
The researchers created RelaCtrl, which figures out which parts of the AI's process are most important for controlling the final image or video. It then focuses more resources on these important parts and less on the less important ones. They also invented a new way to mix information within the AI called the Two-Dimensional Shuffle Mixer, which works faster than previous methods. It's like giving the artist a smart assistant that helps them focus on the most important parts of the painting and use their tools more efficiently.
Why it matters?
This matters because it makes AI-generated images and videos much more efficient to create. RelaCtrl can produce results that are just as good as or better than current methods while using only 15% of the computer power. This could make it easier and cheaper to use AI for creating images and videos, which could be really important for things like making movies, designing products, or creating virtual reality experiences. It's a big step towards making AI art and video creation more practical and accessible.
Abstract
The Diffusion Transformer plays a pivotal role in advancing text-to-image and text-to-video generation, owing primarily to its inherent scalability. However, existing controlled diffusion transformer methods incur significant parameter and computational overheads and suffer from inefficient resource allocation due to their failure to account for the varying relevance of control information across different transformer layers. To address this, we propose the Relevance-Guided Efficient Controllable Generation framework, RelaCtrl, enabling efficient and resource-optimized integration of control signals into the Diffusion Transformer. First, we evaluate the relevance of each layer in the Diffusion Transformer to the control information by assessing the "ControlNet Relevance Score"-i.e., the impact of skipping each control layer on both the quality of generation and the control effectiveness during inference. Based on the strength of the relevance, we then tailor the positioning, parameter scale, and modeling capacity of the control layers to reduce unnecessary parameters and redundant computations. Additionally, to further improve efficiency, we replace the self-attention and FFN in the commonly used copy block with the carefully designed Two-Dimensional Shuffle Mixer (TDSM), enabling efficient implementation of both the token mixer and channel mixer. Both qualitative and quantitative experimental results demonstrate that our approach achieves superior performance with only 15% of the parameters and computational complexity compared to PixArt-delta. More examples are available at https://relactrl.github.io/RelaCtrl/.