MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control
Yuchi Wang, Haiyang Yu, Weikang Bian, Jiefeng Long, Xiao Liang, Chao Feng, Hongsheng Li
2026-04-08
Summary
This paper focuses on improving how well models that understand both images and text, called Multimodal Large Language Models (MLLMs), can create useful connections – or embeddings – between images and text. While these models are good at tasks like finding similar images and descriptions, they haven't fully tapped into their ability to *think* through problems to make even better connections.
What's the problem?
The main challenge is getting these models to actually *reason* when creating embeddings, instead of just mimicking the way reasoning looks. If you simply tell the model to 'think step-by-step,' it might just learn to copy that format without truly understanding the image and text. Also, forcing reasoning on *every* image-text pair is inefficient; sometimes the connection is obvious and doesn't need extra processing, and it can even make things worse by adding noise.
What's the solution?
The researchers developed a new framework called MMEmb-R1. It treats reasoning as something the model can choose to use or not. The model figures out if reasoning will actually help connect the image and text by considering what would happen *if* it didn't reason – a technique called counterfactual intervention. Then, using a method similar to training a game player, the model learns when it's best to reason and when it's better to skip it, saving time and improving accuracy.
Why it matters?
This work is important because it shows how to make these multimodal models more efficient and accurate. MMEmb-R1 achieves top performance on a standard benchmark with a relatively small model size, meaning it can create better image-text connections while using less computing power and responding faster. This is a step towards making these powerful models more practical for real-world applications.
Abstract
MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.