Summary

AnyGPT is a unified multimodal large language model from Fudan University and Shanghai AI Laboratory that can understand and generate text, speech, images and music within a single framework.

Key Features

Any-to-any multimodal model handling text, speech, images and music in one framework
Discrete tokenization unifies all modalities into a single token sequence
AnyInstruct-108k dataset of 108,000 multi-turn multimodal conversations
Strong cross-modal performance in image captioning and speech recognition
Generates images from text and audio from semantic prompts
Stable training without changes to standard LLM architecture
Zero-shot performance comparable to specialised models across modalities
Open research demo and code from Fudan University and Shanghai AI Laboratory

About AnyGPT


It converts every modality into discrete tokens — text, images, audio and music — so one autoregressive transformer can process arbitrary combinations of inputs and outputs without changing the standard LLM architecture. It was trained on AnyInstruct-108k, a dataset of 108,000 multi-turn conversations interleaving modalities, and achieves zero-shot results comparable to specialised models on tasks such as image captioning and speech recognition.


AnyGPT is primarily a research model aimed at developers and researchers exploring multimodal AI, with a public demo and sample code rather than a consumer product. Its any-to-any design suits digital assistants, content creation and interactive media experiments.

Get more likes & reach the top of search results by adding this button on your site!

Embed button preview - Light theme
Embed button preview - Dark theme
TurboType Banner

Subscribe to the AI Search Newsletter

Get top updates in AI to your inbox every weekend. It's free!