Browse / Direct Prompt Injection / AML.CS0040
Hacking ChatGPT's Memories with Prompt Injection
- Technique
- LLM Prompt Crafting
- Also involves
- Indirect Prompt Injection, Jailbreak and Guardrail Bypass, Agent Context and Memory Poisoning
- Target
- OpenAI ChatGPT (Exercise)
How it worked
- AI Attack Adaptation · LLM Prompt Crafting
The researcher crafted a basic prompt asking to set the memory context with a bulleted list of incorrect facts.
- Defense Evasion · LLM Prompt Obfuscation
The researcher placed the prompt in a Google Doc hidden in the header with tiny font matching the document's background color to make it invisible.
- Initial Access · Prompt Infiltration via Public-Facing Application
The Google Doc was shared with the victim, making it accessible to ChatGPT's via its Connected App feature.
- Execution · LLM Prompt Injection: Indirect
When a user referenced something in the shared document, its contents was added to the chat context, and the prompt was executed by ChatGPT.
- Persistence · AI Agent Context Poisoning: Memory
The prompt caused new memories to be introduced, changing the behavior of ChatGPT. The chat window indicated that the memory has been set, despite the lack of human verification or intervention. All future chat sessions will use the poisoned memory store.
- Persistence · Prompt Infiltration via Public-Facing Application
The memory poisoning prompt injection persists in the shared Google Doc, where it can spread to other users and chat sessions, making it difficult to trace sources of the memories and remove.
- Impact · External Harms: User Harm
The victim can be misinformed, misled, or influenced as directed by ChatGPT's poisoned memories.
Summary
Embrace the Red demonstrated that ChatGPT's memory feature is vulnerable to manipulation via prompt injections. To execute the attack, the researcher hid a prompt injection in a shared Google Doc. When a user references the document, its contents is placed into ChatGPT's context via the Connected App feature, and the prompt is executed, poisoning the memory with false facts. The researcher demonstrated that these injected memories persist across chat sessions. Additionally, since the prompt injection payload is introduced through shared resources, this leaves others vulnerable to the same attack and maintains persistence on the system.
References
Source: MITRE ATLAS case studies, used under Apache 2.0.