VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data
Xuefeng Du, Reshmi Ghosh, Robert Sim, Ahmed Salem, Vitor Carvalho, Emily Lawton, Yixuan Li, Jack W. Stokes
2024-10-03
Summary
This paper discusses VLMGuard, a new framework designed to detect and defend vision-language models (VLMs) from harmful prompts that can lead to incorrect or biased outputs.
What's the problem?
Vision-language models are powerful tools that help computers understand both images and text. However, they can be tricked by malicious prompts—inputs that have been intentionally manipulated to produce wrong or harmful responses. Detecting these harmful prompts is essential to ensure the reliability of VLMs, but creating a system to identify them is challenging because there isn't enough labeled data (data that has been marked as good or bad) available for training.
What's the solution?
To solve this problem, the authors developed VLMGuard, which uses unlabeled data from real-world user prompts. This means they can analyze a mix of both good and bad prompts without needing extra human input to label them. They created a scoring system that estimates how likely a prompt is to be harmful, allowing them to train a classifier that can effectively distinguish between benign and malicious prompts. Their experiments show that VLMGuard performs better at detecting harmful inputs compared to existing methods.
Why it matters?
This research is important because it enhances the safety and reliability of AI systems that use vision-language models. By improving the ability to detect malicious prompts, VLMGuard helps ensure that these models provide accurate and trustworthy information, which is crucial for applications in areas like education, healthcare, and customer service.
Abstract
Vision-language models (VLMs) are essential for contextual understanding of both visual and textual information. However, their vulnerability to adversarially manipulated inputs presents significant risks, leading to compromised outputs and raising concerns about the reliability in VLM-integrated applications. Detecting these malicious prompts is thus crucial for maintaining trust in VLM generations. A major challenge in developing a safeguarding prompt classifier is the lack of a large amount of labeled benign and malicious data. To address the issue, we introduce VLMGuard, a novel learning framework that leverages the unlabeled user prompts in the wild for malicious prompt detection. These unlabeled prompts, which naturally arise when VLMs are deployed in the open world, consist of both benign and malicious information. To harness the unlabeled data, we present an automated maliciousness estimation score for distinguishing between benign and malicious samples within this unlabeled mixture, thereby enabling the training of a binary prompt classifier on top. Notably, our framework does not require extra human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiment shows VLMGuard achieves superior detection results, significantly outperforming state-of-the-art methods. Disclaimer: This paper may contain offensive examples; reader discretion is advised.