INCIDENT MAP

Browse / RAG and Vector Store Weaknesses / AML.CS0024

Morris II Worm: RAG-Based Attack

AML.CS0024 · 2024-03-05 case study

Class
RAG and Vector Store Weaknesses
Technique
RAG Poisoning
Also involves
Indirect Prompt Injection, Direct Prompt Injection, Jailbreak and Guardrail Bypass
Target
RAG-based e-mail assistant (Exercise)

How it worked

  1. AI Model Access · AI Model Inference API Access

    The researchers use access to the publicly available GenAI model API that powers the target RAG-based email system.

  2. AI Attack Adaptation · LLM Prompt Crafting

    The researchers construct an adversarial prompt containing instructions to reproduce itself in the model's output and include sensitive information extracted from the supplied context.

  3. AI Attack Adaptation · Verify Attack

    The researchers evaluate the crafted prompts against public GenAI model APIs, checking whether the generated responses reproduce the malicious instructions and include sensitive information from the supplied context.

  4. Initial Access · Prompt Infiltration via Public-Facing Application

    The researchers send an email containing the adversarial self-replicating prompt to an address handled by the target RAG-based email assistant, introducing the malicious instructions into the victim's email environment.

  5. Persistence · RAG Poisoning

    The email application stores the correspondence containing the malicious prompt in its RAG database as part of its normal processing. This contaminates the retrieval corpus, making the instructions available for inclusion in future reply-generation requests.

  6. Execution · LLM Prompt Injection: Triggered

    A subsequent email from another client triggers automatic reply generation. The assistant retrieves the previously stored malicious correspondence as relevant context and includes it in the model request, causing the embedded instructions to influence the response.

  7. Defense Evasion · LLM Jailbreak

    The malicious prompt uses role-play instructions to tell the model to become "Wormy," an email assistant that reproduces the injected instructions and discloses contact information from retrieved correspondence.

  8. Persistence · LLM Prompt Self-Replication

    The self-replicating portion of the prompt causes the generated output to contain the malicious prompt, allowing the worm to propagate.

  9. Collection · Data from AI Services: RAG Databases

    The injected instructions cause the model to extract sensitive information, such as email addresses, physical addresses, and phone numbers, from the correspondence retrieved into the request context and include it in the generated response.

  10. Exfiltration · Exfiltration via Cyber Means

    The email application sends the generated response, containing sensitive information extracted from retrieved correspondence, through its normal automatic email-reply workflow.

  11. Lateral Movement · LLM Prompt Propagation

    The application sends the generated reply containing the reproduced malicious prompt to the client whose email triggered the response, allowing the prompt to propagate to other systems.

  12. Impact · External Harms: User Harm

    Users of the GenAI email assistant may have PII leaked to attackers.

Summary

Researchers developed Morris II, a zero-click worm designed to attack generative AI (GenAI) ecosystems and propagate between connected GenAI systems. The worm uses an adversarial self-replicating prompt which uses prompt injection to replicate the prompt as output and perform malicious activity.

The researchers demonstrate how this worm can propagate through an email system with a RAG-based assistant. They use a target system that automatically ingests received emails, retrieves past correspondences, and generates a reply for the user. To carry out the attack, they send a malicious email containing the adversarial self-replicating prompt, which ends up in the RAG database. The malicious instructions in the prompt tell the assistant to include sensitive user data in the response. Future requests to the email assistant may retrieve the malicious email. This leads to propagation of the worm due to the self-replicating portion of the prompt, as well as leaking private information due to the malicious instructions.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.