Browse / Direct Prompt Injection / AML.CS0021
ChatGPT Conversation Exfiltration
- Technique
- LLM Prompt Crafting
- Target
- OpenAI ChatGPT (Exercise)
How it worked
- AI Attack Adaptation · LLM Prompt Crafting
The researcher developed a prompt that causes ChatGPT to include a Markdown element for an image with the user's conversation embedded in the URL as part of its responses.
- Resource Development · Stage Capabilities
The researcher included the prompt in a webpage, where it could be retrieved by ChatGPT.
- Initial Access · Drive-by Compromise
When the user makes a query that causes ChatGPT to retrieve the webpage using its
WebPilotplugin, it ingests the adversary's prompt. - Execution · LLM Prompt Injection: Indirect
The prompt injection is executed, causing ChatGPT to include a Markdown element for an image hosted on an adversary-controlled server and embed the user's chat history as query parameter in the URL.
- Exfiltration · LLM Response Rendering
ChatGPT automatically renders the image for the user, making the request to the adversary's server for the image contents, and exfiltrating the user's conversation.
- Privilege Escalation · AI Agent Tool Invocation
Additionally, the prompt can cause the LLM to execute other plugins that do not match a user request. In this instance, the researcher demonstrated the
WebPilotplugin making a call to theExpediaplugin. - Impact · External Harms: User Harm
The user's privacy is violated, and they are potentially open to further targeted attacks.
Summary
Embrace the Red demonstrated that ChatGPT users' conversations can be exfiltrated via an indirect prompt injection. To execute the attack, a threat actor uploads a malicious prompt to a public website, where a ChatGPT user may interact with it. The prompt causes ChatGPT to respond with the markdown for an image, whose URL has the user's conversation secretly embedded. ChatGPT renders the image for the user, creating an automatic request to an adversary-controlled script and exfiltrating the user's conversation. Additionally, the researcher demonstrated how the prompt can execute other plugins, opening them up to additional harms.
References
Source: MITRE ATLAS case studies, used under Apache 2.0.