ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers
Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Chen Liang, Tong Shen, Han Zhang, Huanzhang Dou, Yu Liu, Jingren Zhou
2024-12-19
Summary
This paper introduces ChatDiT, a new system that allows users to have interactive conversations with AI about images without needing any extra training. It uses pretrained diffusion transformers to generate images and text together in a natural way.
What's the problem?
Traditional methods for generating images often require extensive training or modifications to the models, which can be time-consuming and complex. This makes it difficult for users to interact with the AI in a simple and intuitive manner.
What's the solution?
ChatDiT solves this problem by using existing pretrained diffusion transformers without any additional tuning or adjustments. It features a multi-agent system that includes components for understanding user instructions, planning actions, and executing those actions to generate images based on conversational prompts. This allows users to create text-image articles, edit images, and more through natural language interactions.
Why it matters?
This research is significant because it simplifies how people can communicate with AI systems about visual content. By eliminating the need for extra training, ChatDiT makes it easier for users to create and manipulate images, which could lead to broader applications in fields like education, entertainment, and design.
Abstract
Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the inherent in-context generation capabilities of pretrained diffusion transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks with minimal or no architectural modifications. These capabilities are unlocked by concatenating self-attention tokens across multiple input and target images, combined with grouped and masked generation pipelines. Building upon this foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive visual generation framework that leverages pretrained diffusion transformers in their original form, requiring no additional tuning, adapters, or modifications. Users can interact with ChatDiT to create interleaved text-image articles, multi-page picture books, edit images, design IP derivatives, or develop character design settings, all through free-form natural language across one or more conversational rounds. At its core, ChatDiT employs a multi-agent system comprising three key components: an Instruction-Parsing agent that interprets user-uploaded images and instructions, a Strategy-Planning agent that devises single-step or multi-step generation actions, and an Execution agent that performs these actions using an in-context toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with diverse instructions and varying numbers of input and target images. Despite its simplicity and training-free approach, ChatDiT surpasses all competitors, including those specifically designed and trained on extensive multi-task datasets. We further identify key limitations of pretrained DiTs in zero-shot adapting to tasks. We release all code, agents, results, and intermediate outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT