INCIDENT MAP

Browse / Jailbreak and Guardrail Bypass / AML.CS0057

Storm-2139 Azure OpenAI Guardrail Bypass

AML.CS0057 · 2024-07-01 case study

Class
Jailbreak and Guardrail Bypass
Technique
LLM Jailbreak
Also involves
Misinformation and External Harm, AI Framework and Platform Flaws
Target
Microsoft Azure OpenAI Service (Incident)

How it worked

  1. Resource Development · Develop Capabilities

    Storm-2139 creators developed a tool called de3u to facilitate unauthorized use of generative AI services and bypass safeguards.

  2. Resource Development · Acquire Infrastructure

    Storm-2139 acquired infrastructure to support a service that sold access to the jailbroken Azure OpenAI Service.

  3. Resource Development · Stage Capabilities

    Storm-2139 staged and operated a reverse proxy service to allow other malicious users to interact with abused generative AI services.

  4. Initial Access · Valid Accounts

    Storm-2139 used exposed customer credentials scraped from public sources to access valid accounts for generative AI services.

  5. AI Model Access · AI Model Inference API Access

    The stolen credentials provided access Azure OpenAI Service, allowing the actors and their customers to submit prompts and generate content. Storm-2139's de3u tool was used as the frontend for this access.

  6. Defense Evasion · LLM Jailbreak

    Storm-2139 deliberately bypassed Azure OpenAI Service safeguards and content filters to generate prohibited outputs. Microsoft reported that the actors iterated on blocked prompts, substituted celebrity descriptions, and used altered wording or technical notation to evade filters.

  7. AI Attack Adaptation · Generate Deepfakes

    End users generated abusive synthetic imagery, including non-consensual intimate images of celebrities and other sexually explicit, misogynistic, violent, or hateful content.

  8. Impact · External Harms: Societal Harm

    The generated abusive imagery could cause direct harm to depicted individuals.

  9. Impact · External Harms: Financial Harm

    Users whose accounts were stolen were harmed financially.

Summary

Storm-2139 built custom jailbreak tooling to bypass guardrails on Azure OpenAI Services, allowing users to generate harmful synthetic content.

Microsoft reported that members of Storm-2139 scraped exposed customer credentials from public sources and used them to access accounts for generative AI services. The group developed and operated tools and services that bypassed safety safeguards, modified service capabilities, and enabled end users to generate harmful and illicit content, including non-consensual intimate images of celebrities and other sexually explicit content.

The operation included creators who developed illicit tools, providers who modified and supplied those tools, and users who generated prohibited content. Microsoft pursued civil legal action.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.