Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads
Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan
2025-11-11
Summary
This paper introduces a new way to check if the steps a large language model (LLM) takes while solving a problem are correct, without needing a lot of computing power or human help.
What's the problem?
LLMs are getting better at tackling complicated problems by breaking them down into smaller steps, but it's hard to know if each step is actually right. Existing methods to verify these steps, like 'Process Reward Models', are either too slow, only work in specific areas, or require tons of labeled data created by people or other models.
What's the solution?
The researchers developed a system called 'UHeads' that's attached to an existing LLM. UHeads looks at the LLM's internal workings *while* it's thinking and estimates how confident the LLM is in each step. This confidence score acts as a check – if the LLM is unsure, the step might be wrong. Importantly, UHeads learns what 'uncertainty' looks like either by getting feedback from a more powerful LLM or by learning from the original LLM itself, making it automatic and efficient. It's also very small in terms of the number of parameters it uses.
Why it matters?
This work shows that LLMs actually 'know' when they're unsure about their reasoning, and we can tap into that knowledge to improve their accuracy and make their thought process more transparent. Because UHeads is lightweight and doesn't need a lot of extra data, it's a practical step towards building LLMs that can reliably explain *how* they arrive at an answer.
Abstract
Solving complex tasks usually requires LLMs to generate long multi-step reasoning chains. Previous work has shown that verifying the correctness of individual reasoning steps can further improve the performance and efficiency of LLMs on such tasks and enhance solution interpretability. However, existing verification approaches, such as Process Reward Models (PRMs), are either computationally expensive, limited to specific domains, or require large-scale human or model-generated annotations. Thus, we propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty scores. We train transformer-based uncertainty quantification heads (UHeads) that use the internal states of a frozen LLM to estimate the uncertainty of its reasoning steps during generation. The approach is fully automatic: target labels are generated either by another larger LLM (e.g., DeepSeek R1) or in a self-supervised manner by the original model itself. UHeads are both effective and lightweight, containing less than 10M parameters. Across multiple domains, including mathematics, planning, and general knowledge question answering, they match or even surpass the performance of PRMs that are up to 810x larger. Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification, offering a promising direction toward scalable and generalizable introspective LLMs.