Browse / Jailbreak and Guardrail Bypass / AML.CS0057
Storm-2139 Azure OpenAI Guardrail Bypass
- Technique
- LLM Jailbreak
- Target
- Microsoft Azure OpenAI Service (Incident)
How it worked
- Resource Development · Develop Capabilities
Storm-2139 creators developed a tool called de3u to facilitate unauthorized use of generative AI services and bypass safeguards.
- Resource Development · Acquire Infrastructure
Storm-2139 acquired infrastructure to support a service that sold access to the jailbroken Azure OpenAI Service.
- Resource Development · Stage Capabilities
Storm-2139 staged and operated a reverse proxy service to allow other malicious users to interact with abused generative AI services.
- Initial Access · Valid Accounts
Storm-2139 used exposed customer credentials scraped from public sources to access valid accounts for generative AI services.
- AI Model Access · AI Model Inference API Access
The stolen credentials provided access Azure OpenAI Service, allowing the actors and their customers to submit prompts and generate content. Storm-2139's de3u tool was used as the frontend for this access.
- Defense Evasion · LLM Jailbreak
Storm-2139 deliberately bypassed Azure OpenAI Service safeguards and content filters to generate prohibited outputs. Microsoft reported that the actors iterated on blocked prompts, substituted celebrity descriptions, and used altered wording or technical notation to evade filters.
- AI Attack Adaptation · Generate Deepfakes
End users generated abusive synthetic imagery, including non-consensual intimate images of celebrities and other sexually explicit, misogynistic, violent, or hateful content.
- Impact · External Harms: Societal Harm
The generated abusive imagery could cause direct harm to depicted individuals.
- Impact · External Harms: Financial Harm
Users whose accounts were stolen were harmed financially.
Summary
Storm-2139 built custom jailbreak tooling to bypass guardrails on Azure OpenAI Services, allowing users to generate harmful synthetic content.
Microsoft reported that members of Storm-2139 scraped exposed customer credentials from public sources and used them to access accounts for generative AI services. The group developed and operated tools and services that bypassed safety safeguards, modified service capabilities, and enabled end users to generate harmful and illicit content, including non-consensual intimate images of celebrities and other sexually explicit content.
The operation included creators who developed illicit tools, providers who modified and supplied those tools, and users who generated prohibited content. Microsoft pursued civil legal action.
References
- https://blogs.microsoft.com/on-the-issues/2025/01/10/taking-legal-action-to-protect-the-public-from-abusive-ai-generated-content/
- https://blogs.microsoft.com/on-the-issues/2025/02/27/disrupting-cybercrime-abusing-gen-ai/
- https://news.microsoft.com/source/features/ai/how-microsoft-is-taking-down-ai-hackers-who-create-harmful-images-of-celebrities-and-others/
- https://incidentdatabase.ai/cite/955/
Source: MITRE ATLAS case studies, used under Apache 2.0.