Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation
Jinyi Han, Tingyun Li, Shisong Chen, Jie Shi, Xinyi Wang, Guanglei Yue, Jiaqing Liang, Xin Lin, Liqian Wen, Zulong Chen, Yanghua Xiao
2025-08-20
Summary
This paper presents a new way to measure how confident a large language model is when it generates text, called FineCE. It aims to make these confidence scores more accurate and detailed than previous methods, helping to make AI more trustworthy.
What's the problem?
Even though AI language models are really good at many tasks, they don't really understand themselves and often act super sure of themselves even when they're wrong. This means we can't always trust what they say because their confidence scores aren't reliable or detailed enough to show how sure they are at every step of writing something.
What's the solution?
The researchers created a method called FineCE. They built a special system to gather data that shows how language models usually respond. Then, they trained a model to predict confidence scores for any piece of text. They also developed a technique called Backward Confidence Integration, which uses what comes after a piece of text to make the confidence score for that text better. Plus, they figured out the best places to check confidence while the AI is generating text.
Why it matters?
Having accurate, fine-grained confidence scores is super important because it makes AI language models more dependable and trustworthy. If we know when an AI is unsure, we can avoid relying on potentially incorrect information, which is crucial for many real-world applications.
Abstract
While large language models (LLMs) have demonstrated remarkable performance across diverse tasks, they fundamentally lack self-awareness and frequently exhibit overconfidence, assigning high confidence scores to incorrect predictions. Accurate confidence estimation is therefore critical for enhancing the trustworthiness and reliability of LLM-generated outputs. However, existing approaches suffer from coarse-grained scoring mechanisms that fail to provide fine-grained, continuous confidence estimates throughout the generation process. To address these limitations, we introduce FineCE, a novel confidence estimation method that delivers accurate, fine-grained confidence scores during text generation. Specifically, we first develop a comprehensive pipeline for constructing training data that effectively captures the underlying probabilistic distribution of LLM responses, and then train a model to predict confidence scores for arbitrary text sequences in a supervised manner. Furthermore, we propose a Backward Confidence Integration (BCI) strategy that leverages information from the subsequent text to enhance confidence estimation for the current sequence during inference. We also introduce three strategies for identifying optimal positions to perform confidence estimation within the generation process. Extensive experiments on multiple benchmark datasets demonstrate that FineCE consistently outperforms existing classical confidence estimation methods. Our code and all baselines used in the paper are available on GitHub.