In-Context LoRA for Diffusion Transformers
Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, Jingren Zhou
2024-11-04
Summary
This paper introduces a new approach called In-Context LoRA (IC-LoRA) for improving image generation using diffusion transformers. It combines techniques from different types of models to create high-quality images more efficiently.
What's the problem?
While diffusion transformers have shown promise in generating images, they often produce lower-quality results than expected, even with significant computing power. Existing methods typically rely on complex training processes that may not fully utilize the models' capabilities.
What's the solution?
The authors propose a simpler method that leverages the inherent abilities of text-to-image diffusion transformers for in-context generation. They suggest a new training pipeline that involves concatenating images instead of tokens, performing joint captioning for multiple images, and applying task-specific tuning with small datasets. This approach allows the model to generate better images without needing extensive modifications or large amounts of training data.
Why it matters?
This research is important because it enhances the way AI generates images, making it more efficient and effective. By simplifying the training process and improving image quality, IC-LoRA can benefit various applications in art, design, and content creation. Additionally, by sharing their code and models, the authors encourage further exploration and innovation in the field of generative AI.
Abstract
Recent research arXiv:2410.15027 has explored the use of diffusion transformers (DiTs) for task-agnostic image generation by simply concatenating attention tokens across images. However, despite substantial computational resources, the fidelity of the generated images remains suboptimal. In this study, we reevaluate and streamline this framework by hypothesizing that text-to-image DiTs inherently possess in-context generation capabilities, requiring only minimal tuning to activate them. Through diverse task experiments, we qualitatively demonstrate that existing text-to-image DiTs can effectively perform in-context generation without any tuning. Building on this insight, we propose a remarkably simple pipeline to leverage the in-context abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint captioning of multiple images, and (3) apply task-specific LoRA tuning using small datasets (e.g., 20sim 100 samples) instead of full-parameter tuning with large datasets. We name our models In-Context LoRA (IC-LoRA). This approach requires no modifications to the original DiT models, only changes to the training data. Remarkably, our pipeline generates high-fidelity image sets that better adhere to prompts. While task-specific in terms of tuning data, our framework remains task-agnostic in architecture and pipeline, offering a powerful tool for the community and providing valuable insights for further research on product-level task-agnostic generation systems. We release our code, data, and models at https://github.com/ali-vilab/In-Context-LoRA