Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models
Sina Tayebati, Divake Kumar, Nastaran Darabi, Dinithi Jayasuriya, Ranganath Krishnan, Amit Ranjan Trivedi
2025-02-12
Summary
This paper talks about a new way to make AI models better at knowing when they're unsure, especially for important tasks where mistakes could be dangerous. The researchers created a system that helps AI decide when to answer questions and when to say it's not sure, adapting to different situations.
What's the problem?
Big AI models that work with text and images are being used for important jobs, but it's hard to tell when they might be wrong. Current methods to check their uncertainty don't change based on how hard the task is or how different the new information might be from what the AI learned before. This can lead to the AI being overconfident or too cautious.
What's the solution?
The researchers came up with a new method called 'learnable conformal abstention.' It uses a type of AI learning called reinforcement learning to constantly adjust how the model decides when it's sure enough to answer. This new method looks at multiple factors to balance being accurate, covering enough information, and giving useful answers. They tested it on many different AI models and types of tasks.
Why it matters?
This matters because as we use AI for more important decisions, we need to trust that it knows when it might be wrong. The new method makes AI models better at recognizing when they might make mistakes, which is crucial for using them safely in areas like healthcare or self-driving cars. It helps the AI give more reliable answers and know when to ask for human help, making it safer and more trustworthy for real-world use.
Abstract
Large Language and Vision-Language Models (LLMs/VLMs) are increasingly used in safety-critical applications, yet their opaque decision-making complicates risk assessment and reliability. Uncertainty quantification (UQ) helps assess prediction confidence and enables abstention when uncertainty is high. Conformal prediction (CP), a leading UQ method, provides statistical guarantees but relies on static thresholds, which fail to adapt to task complexity and evolving data distributions, leading to suboptimal trade-offs in accuracy, coverage, and informativeness. To address this, we propose learnable conformal abstention, integrating reinforcement learning (RL) with CP to optimize abstention thresholds dynamically. By treating CP thresholds as adaptive actions, our approach balances multiple objectives, minimizing prediction set size while maintaining reliable coverage. Extensive evaluations across diverse LLM/VLM benchmarks show our method outperforms Least Ambiguous Classifiers (LAC) and Adaptive Prediction Sets (APS), improving accuracy by up to 3.2%, boosting AUROC for hallucination detection by 22.19%, enhancing uncertainty-guided selective generation (AUARC) by 21.17%, and reducing calibration error by 70%-85%. These improvements hold across multiple models and datasets while consistently meeting the 90% coverage target, establishing our approach as a more effective and flexible solution for reliable decision-making in safety-critical applications. The code is available at: {https://github.com/sinatayebati/vlm-uncertainty}.