The Flexibility Trap: Why Arbitrary Order Limits Reasoning Potential in Diffusion Language Models
Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, Gao Huang
2026-01-23
Summary
This paper investigates Diffusion Large Language Models (dLLMs), which are a newer type of language model that can generate text in any order, unlike traditional models that go strictly from left to right. The research challenges the idea that this flexibility automatically makes dLLMs better at complex tasks like math and coding.
What's the problem?
The core issue is that while dLLMs *can* generate text in any order, they often don't use this ability to actually explore different solutions to a problem. Instead, they tend to avoid parts of the problem that are uncertain, quickly settling on an answer without fully considering all possibilities. This limits their reasoning ability, even though the flexibility *should* allow for more thorough exploration. Existing methods trying to improve dLLMs using this flexibility are complicated and may be focusing on the wrong thing.
What's the solution?
The researchers found that surprisingly, forcing the dLLM to generate text in a standard, left-to-right order actually leads to better results. They used a relatively simple technique called Group Relative Policy Optimization (GRPO), which is commonly used with traditional language models, and named their approach 'JustGRPO'. This method still allows for fast, parallel processing like dLLMs are known for, but it avoids the pitfalls of uncontrolled, arbitrary order generation.
Why it matters?
This work is important because it shows that simply adding flexibility to a language model doesn't automatically make it smarter. It suggests that the way a model *uses* its abilities is more crucial than the abilities themselves. It also simplifies the process of training dLLMs for reasoning tasks, as it removes the need for complex methods designed to manage arbitrary order generation, and still achieves high accuracy on challenging benchmarks like math problems.
Abstract
Diffusion Large Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs, enabling token generation in arbitrary orders. Intuitively, this flexibility implies a solution space that strictly supersets the fixed autoregressive trajectory, theoretically unlocking superior reasoning potential for general tasks like mathematics and coding. Consequently, numerous works have leveraged reinforcement learning (RL) to elicit the reasoning capability of dLLMs. In this paper, we reveal a counter-intuitive reality: arbitrary order generation, in its current form, narrows rather than expands the reasoning boundary of dLLMs. We find that dLLMs tend to exploit this order flexibility to bypass high-uncertainty tokens that are crucial for exploration, leading to a premature collapse of the solution space. This observation challenges the premise of existing RL approaches for dLLMs, where considerable complexities, such as handling combinatorial trajectories and intractable likelihoods, are often devoted to preserving this flexibility. We demonstrate that effective reasoning is better elicited by intentionally forgoing arbitrary order and applying standard Group Relative Policy Optimization (GRPO) instead. Our approach, JustGRPO, is minimalist yet surprisingly effective (e.g., 89.1% accuracy on GSM8K) while fully retaining the parallel decoding ability of dLLMs. Project page: https://nzl-thu.github.io/the-flexibility-trap