INCIDENT MAP

Browse / Direct Prompt Injection / AML.CS0021

ChatGPT Conversation Exfiltration

AML.CS0021 · 2023-05-01 case study

Class
Direct Prompt Injection
Technique
LLM Prompt Crafting
Also involves
Indirect Prompt Injection, Improper Output Handling, Tool and Agent Misuse
Target
OpenAI ChatGPT (Exercise)

How it worked

  1. AI Attack Adaptation · LLM Prompt Crafting

    The researcher developed a prompt that causes ChatGPT to include a Markdown element for an image with the user's conversation embedded in the URL as part of its responses.

  2. Resource Development · Stage Capabilities

    The researcher included the prompt in a webpage, where it could be retrieved by ChatGPT.

  3. Initial Access · Drive-by Compromise

    When the user makes a query that causes ChatGPT to retrieve the webpage using its WebPilot plugin, it ingests the adversary's prompt.

  4. Execution · LLM Prompt Injection: Indirect

    The prompt injection is executed, causing ChatGPT to include a Markdown element for an image hosted on an adversary-controlled server and embed the user's chat history as query parameter in the URL.

  5. Exfiltration · LLM Response Rendering

    ChatGPT automatically renders the image for the user, making the request to the adversary's server for the image contents, and exfiltrating the user's conversation.

  6. Privilege Escalation · AI Agent Tool Invocation

    Additionally, the prompt can cause the LLM to execute other plugins that do not match a user request. In this instance, the researcher demonstrated the WebPilot plugin making a call to the Expedia plugin.

  7. Impact · External Harms: User Harm

    The user's privacy is violated, and they are potentially open to further targeted attacks.

Summary

Embrace the Red demonstrated that ChatGPT users' conversations can be exfiltrated via an indirect prompt injection. To execute the attack, a threat actor uploads a malicious prompt to a public website, where a ChatGPT user may interact with it. The prompt causes ChatGPT to respond with the markdown for an image, whose URL has the user's conversation secretly embedded. ChatGPT renders the image for the user, creating an automatic request to an adversary-controlled script and exfiltrating the user's conversation. Additionally, the researcher demonstrated how the prompt can execute other plugins, opening them up to additional harms.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.