CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing
Jianxiong Gao, Yichang Liu, Baofeng Yang, Jianfeng Feng, Yanwei Fu
2025-03-12
Summary
This paper talks about CineBrain, a brain-scan dataset that records people watching TV shows while wearing EEG caps and MRI scanners to study how brains process stories and sounds.
What's the problem?
Scientists struggle to study how brains handle real-world experiences like watching movies because combining high-speed brainwave data (EEG) and detailed brain images (fMRI) is tricky and lacks good datasets.
What's the solution?
CineBrain records both EEG and fMRI data while people watch TV episodes, then uses a new AI tool (CineSync) to merge these signals and recreate what they watched from brain activity alone.
Why it matters?
This helps researchers understand how brains process complex stories and sounds in real time, which could improve brain-computer interfaces or treatments for brain disorders.
Abstract
In this paper, we introduce CineBrain, the first large-scale dataset featuring simultaneous EEG and fMRI recordings during dynamic audiovisual stimulation. Recognizing the complementary strengths of EEG's high temporal resolution and fMRI's deep-brain spatial coverage, CineBrain provides approximately six hours of narrative-driven content from the popular television series The Big Bang Theory for each of six participants. Building upon this unique dataset, we propose CineSync, an innovative multimodal decoding framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural Latent Decoder. Our approach effectively fuses EEG and fMRI signals, significantly improving the reconstruction quality of complex audiovisual stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a comprehensive evaluation protocol that assesses reconstructions across semantic and perceptual dimensions. Experimental results demonstrate that CineSync achieves state-of-the-art video reconstruction performance and highlight our initial success in combining fMRI and EEG for reconstructing both video and audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.