Hallucinating AI Hijacking Attack: Large Language Models and Malicious Code Recommenders
David Noever, Forrest McKee
2024-10-10
Summary
This paper discusses a new type of cyberattack called the Hallucinating AI Hijacking Attack, which exploits large language models (LLMs) to generate harmful code recommendations by manipulating context.
What's the problem?
As LLMs become more advanced and widely used, there are growing concerns about their potential misuse, especially in generating malicious code. Current safeguards in these models can sometimes fail when the context of a request changes, allowing attackers to trick the model into providing harmful suggestions or code snippets.
What's the solution?
The authors investigate how LLMs can be manipulated to produce dangerous outputs by changing the context of user prompts. They demonstrate that while LLMs may refuse to provide harmful suggestions when directly asked, they can be led to drop their guard when asked to solve programming challenges. The research includes examples from popular code repositories and shows how attackers can use these loopholes to propose malicious actions disguised as helpful recommendations.
Why it matters?
This research is important because it highlights vulnerabilities in LLMs that could be exploited for malicious purposes. By understanding these weaknesses, developers can improve safety measures and ensure that AI systems are better at recognizing and rejecting harmful requests, ultimately protecting users and maintaining the integrity of AI applications.
Abstract
The research builds and evaluates the adversarial potential to introduce copied code or hallucinated AI recommendations for malicious code in popular code repositories. While foundational large language models (LLMs) from OpenAI, Google, and Anthropic guard against both harmful behaviors and toxic strings, previous work on math solutions that embed harmful prompts demonstrate that the guardrails may differ between expert contexts. These loopholes would appear in mixture of expert's models when the context of the question changes and may offer fewer malicious training examples to filter toxic comments or recommended offensive actions. The present work demonstrates that foundational models may refuse to propose destructive actions correctly when prompted overtly but may unfortunately drop their guard when presented with a sudden change of context, like solving a computer programming challenge. We show empirical examples with trojan-hosting repositories like GitHub, NPM, NuGet, and popular content delivery networks (CDN) like jsDelivr which amplify the attack surface. In the LLM's directives to be helpful, example recommendations propose application programming interface (API) endpoints which a determined domain-squatter could acquire and setup attack mobile infrastructure that triggers from the naively copied code. We compare this attack to previous work on context-shifting and contrast the attack surface as a novel version of "living off the land" attacks in the malware literature. In the latter case, foundational language models can hijack otherwise innocent user prompts to recommend actions that violate their owners' safety policies when posed directly without the accompanying coding support request.