MPJudge: Towards Perceptual Assessment of Music-Induced Paintings
Shiqi Jiang, Tianyi Liang, Changbo Wang, Chenhui Li
2025-11-11
Summary
This paper focuses on how to automatically tell if a painting was actually inspired by a specific piece of music, and how well it 'matches' that music.
What's the problem?
Currently, computer programs trying to judge this match rely on figuring out the *emotions* in both the music and the painting. The problem is that emotions are subjective and can be misinterpreted, leading to inaccurate assessments. Plus, a painting's connection to music isn't just about feeling – it's about how visual elements relate to musical qualities beyond just emotion.
What's the solution?
The researchers created a new system called MPJudge. First, they built a large collection of paintings and the music that inspired them, and had art experts rate how well each painting visually connected to its music. This created a dataset called MPD. Then, they designed MPJudge to directly compare features of the music and the painting, using a technique that blends musical information into how the painting is 'read' by the computer. They also used a special training method to help the system learn even when the connections aren't obvious.
Why it matters?
This work is important because it provides a more accurate way to evaluate music-induced paintings. This could be useful for artists, art historians, or even for developing AI that can create art inspired by music. By focusing on perceptual coherence – how the painting and music relate visually and aurally – instead of just emotion, the system offers a more nuanced and reliable assessment.
Abstract
Music induced painting is a unique artistic practice, where visual artworks are created under the influence of music. Evaluating whether a painting faithfully reflects the music that inspired it poses a challenging perceptual assessment task. Existing methods primarily rely on emotion recognition models to assess the similarity between music and painting, but such models introduce considerable noise and overlook broader perceptual cues beyond emotion. To address these limitations, we propose a novel framework for music induced painting assessment that directly models perceptual coherence between music and visual art. We introduce MPD, the first large scale dataset of music painting pairs annotated by domain experts based on perceptual coherence. To better handle ambiguous cases, we further collect pairwise preference annotations. Building on this dataset, we present MPJudge, a model that integrates music features into a visual encoder via a modulation based fusion mechanism. To effectively learn from ambiguous cases, we adopt Direct Preference Optimization for training. Extensive experiments demonstrate that our method outperforms existing approaches. Qualitative results further show that our model more accurately identifies music relevant regions in paintings.