MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization
Xiangyu Zhao, Junming Lin, Tianhao Liang, Yifan Zhou, Wenhao Chai, Yuzhe Gu, Weiyun Wang, Kai Chen, Gen Luo, Wenwei Zhang, Junchi Yan, Hua Yang, Haodong Duan, Xue Yang
2025-10-10
Summary
This paper investigates how well current image and text understanding AI models, called Multimodal Large Language Models or MLLMs, can handle complex problems that require thinking through multiple steps and correcting mistakes along the way – what researchers call 'long-chain reflective reasoning'. They found these models struggle with this type of reasoning and then developed new techniques to improve their ability to do so.
What's the problem?
While MLLMs are good at basic reasoning like math and logic, they aren't very good at tackling problems that require a lot of back-and-forth thinking, like planning a complex project or debugging a difficult code. The issue is that these models often get stuck and can't recover from errors when a problem requires many steps to solve. There wasn't a good way to measure this weakness, and existing training methods weren't helping them improve.
What's the solution?
The researchers created a new challenging test, called MM-HELIX, with over a thousand problems specifically designed to test this 'reflective reasoning' ability. They then built a large dataset of example solutions to these problems, showing the step-by-step thinking process. Finally, they developed a new training method called Adaptive Hybrid Policy Optimization (AHPO) that combines learning from these examples with the model learning on its own through trial and error, allowing it to learn even when it doesn't get clear feedback. They applied this to an existing model, Qwen2.5-VL-7B, and saw significant improvements.
Why it matters?
This work is important because it shows that MLLMs *can* be taught to think more deeply and solve complex problems that require multiple steps and corrections. By identifying this weakness and developing a way to address it, the researchers are paving the way for more powerful and versatile AI models that can handle real-world challenges more effectively, going beyond just simple question answering or basic calculations.
Abstract
While current Multimodal Large Language Models (MLLMs) have demonstrated proficiency in reasoning tasks such as mathematics and logic, their capacity for long-chain reflective reasoning, a prerequisite for solving complex real-world problems, remains largely underexplored. In this work, we first conduct an extensive empirical investigation to evaluate this capability. Leveraging a carefully designed data synthesis engine, we construct MM-HELIX, a multimodal benchmark consisting 1,260 samples of 42 challenging synthetic tasks that require iterative thinking and backtracking. Empirical results on this benchmark reveal that existing MLLMs exhibit significant performance deficits in long-chain reflective reasoning. To address this limitation, we generate post-training data and further explore learning paradigms for exploiting such data. We first develop the Step-Elicited Response Generation pipeline to create MM-HELIX-100K, a large-scale dataset of 100k high-quality, reflective reasoning traces for instruction-tuning stage. Given that standard Reinforcement Learning fails on complex tasks due to sparse reward signals and catastrophic forgetting after Supervised Fine-Tuning, we propose Adaptive Hybrid Policy Optimization (AHPO), a novel training strategy that dynamically unifies offline supervision and online optimization into a single stage. This strategy enables the model to learn from expert data when rewards are sparse and conduct independent exploration once proficient. When applied to the Qwen2.5-VL-7B baseline, our method achieves a +18.6\% accuracy improvement on MM-HELIX benchmark and demonstrates strong generalization with a +5.7\% average performance gain on general mathematic and logic tasks. Our work demonstrate that reflective reasoning in MLLMs can be effectively learned and generalized, paving the way for developing more capable MLLMs.