INCIDENT MAP

Browse / Direct Prompt Injection / AML.CS0040

Hacking ChatGPT's Memories with Prompt Injection

AML.CS0040 · 2024-02-01 case study

Class
Direct Prompt Injection
Technique
LLM Prompt Crafting
Also involves
Indirect Prompt Injection, Jailbreak and Guardrail Bypass, Agent Context and Memory Poisoning
Target
OpenAI ChatGPT (Exercise)

How it worked

  1. AI Attack Adaptation · LLM Prompt Crafting

    The researcher crafted a basic prompt asking to set the memory context with a bulleted list of incorrect facts.

  2. Defense Evasion · LLM Prompt Obfuscation

    The researcher placed the prompt in a Google Doc hidden in the header with tiny font matching the document's background color to make it invisible.

  3. Initial Access · Prompt Infiltration via Public-Facing Application

    The Google Doc was shared with the victim, making it accessible to ChatGPT's via its Connected App feature.

  4. Execution · LLM Prompt Injection: Indirect

    When a user referenced something in the shared document, its contents was added to the chat context, and the prompt was executed by ChatGPT.

  5. Persistence · AI Agent Context Poisoning: Memory

    The prompt caused new memories to be introduced, changing the behavior of ChatGPT. The chat window indicated that the memory has been set, despite the lack of human verification or intervention. All future chat sessions will use the poisoned memory store.

  6. Persistence · Prompt Infiltration via Public-Facing Application

    The memory poisoning prompt injection persists in the shared Google Doc, where it can spread to other users and chat sessions, making it difficult to trace sources of the memories and remove.

  7. Impact · External Harms: User Harm

    The victim can be misinformed, misled, or influenced as directed by ChatGPT's poisoned memories.

Summary

Embrace the Red demonstrated that ChatGPT's memory feature is vulnerable to manipulation via prompt injections. To execute the attack, the researcher hid a prompt injection in a shared Google Doc. When a user references the document, its contents is placed into ChatGPT's context via the Connected App feature, and the prompt is executed, poisoning the memory with false facts. The researcher demonstrated that these injected memories persist across chat sessions. Additionally, since the prompt injection payload is introduced through shared resources, this leaves others vulnerable to the same attack and maintains persistence on the system.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.