Browse / RAG and Vector Store Weaknesses / AML.CS0024
Morris II Worm: RAG-Based Attack
- Technique
- RAG Poisoning
- Target
- RAG-based e-mail assistant (Exercise)
How it worked
- AI Model Access · AI Model Inference API Access
The researchers use access to the publicly available GenAI model API that powers the target RAG-based email system.
- AI Attack Adaptation · LLM Prompt Crafting
The researchers construct an adversarial prompt containing instructions to reproduce itself in the model's output and include sensitive information extracted from the supplied context.
- AI Attack Adaptation · Verify Attack
The researchers evaluate the crafted prompts against public GenAI model APIs, checking whether the generated responses reproduce the malicious instructions and include sensitive information from the supplied context.
- Initial Access · Prompt Infiltration via Public-Facing Application
The researchers send an email containing the adversarial self-replicating prompt to an address handled by the target RAG-based email assistant, introducing the malicious instructions into the victim's email environment.
- Persistence · RAG Poisoning
The email application stores the correspondence containing the malicious prompt in its RAG database as part of its normal processing. This contaminates the retrieval corpus, making the instructions available for inclusion in future reply-generation requests.
- Execution · LLM Prompt Injection: Triggered
A subsequent email from another client triggers automatic reply generation. The assistant retrieves the previously stored malicious correspondence as relevant context and includes it in the model request, causing the embedded instructions to influence the response.
- Defense Evasion · LLM Jailbreak
The malicious prompt uses role-play instructions to tell the model to become "Wormy," an email assistant that reproduces the injected instructions and discloses contact information from retrieved correspondence.
- Persistence · LLM Prompt Self-Replication
The self-replicating portion of the prompt causes the generated output to contain the malicious prompt, allowing the worm to propagate.
- Collection · Data from AI Services: RAG Databases
The injected instructions cause the model to extract sensitive information, such as email addresses, physical addresses, and phone numbers, from the correspondence retrieved into the request context and include it in the generated response.
- Exfiltration · Exfiltration via Cyber Means
The email application sends the generated response, containing sensitive information extracted from retrieved correspondence, through its normal automatic email-reply workflow.
- Lateral Movement · LLM Prompt Propagation
The application sends the generated reply containing the reproduced malicious prompt to the client whose email triggered the response, allowing the prompt to propagate to other systems.
- Impact · External Harms: User Harm
Users of the GenAI email assistant may have PII leaked to attackers.
Summary
Researchers developed Morris II, a zero-click worm designed to attack generative AI (GenAI) ecosystems and propagate between connected GenAI systems. The worm uses an adversarial self-replicating prompt which uses prompt injection to replicate the prompt as output and perform malicious activity.
The researchers demonstrate how this worm can propagate through an email system with a RAG-based assistant. They use a target system that automatically ingests received emails, retrieves past correspondences, and generates a reply for the user. To carry out the attack, they send a malicious email containing the adversarial self-replicating prompt, which ends up in the RAG database. The malicious instructions in the prompt tell the assistant to include sensitive user data in the response. Future requests to the email assistant may retrieve the malicious email. This leads to propagation of the worm due to the self-replicating portion of the prompt, as well as leaking private information due to the malicious instructions.
References
Source: MITRE ATLAS case studies, used under Apache 2.0.