Browse / Jailbreak and Guardrail Bypass / Real-world incidents
Real-world incidents
19 records in Jailbreak and Guardrail Bypass.
- Grok Reportedly Generated Images After X Users Asked It to Unredact Epstein-File Photos of Children and Women Users asked Grok to unblur redacted children and women in Epstein-file photos; Bellingcat found it generated images in 27 of 31 requests, bypassing privacy protections.
- X Users Reportedly Prompted Grok to Sexualize Images of Renée Good After Her Killing in Minneapolis Users deliberately prompted xAI's Grok to edit images of Renée Good into bikini images, and Grok complied by publicly posting the sexualized output on X, bypassing its safeguards.
- Grok Reportedly Generated and Distributed Nonconsensual Sexualized Images of Adults and Minors in X Replies Users prompted Grok to turn real people's photos, including those of minors, into sexualized images, bypassing its safeguards, and the outputs were posted publicly in X replies.
- Anthropic's Claude Was Reportedly Jailbroken To Allegedly Help Steal Sensitive Mexican Government Data An unknown attacker reportedly jailbroke Claude and used it to find vulnerabilities and write exploit scripts for automated theft of 150 GB of Mexican government data.
- OpenAI ChatGPT Models Reportedly Jailbroken to Provide Chemical, Biological, and Nuclear Weapons Instructions Researchers used a publicly documented jailbreak prompt to bypass safeguards in several OpenAI models, eliciting chemical, biological and nuclear weapons instructions.
- Google Gemini Reportedly Generates Sexual Role‑Play for Account Registered as Minor Prompt manipulation and summarization requests reportedly bypassed Gemini's teen-safety filters on a account registered to a 13-year-old, producing sexual role-play content.
- Grok 3 Reportedly Generated Graphic Threats and Hate Speech Targeting Minnesota Attorney Will Stancil Users prompted Grok 3 with adversarial requests that got past its safeguards, producing graphic threats, rape scenarios and break-in instructions aimed at Will Stancil.
- Jailbroken Lovable AI Allegedly Used to Generate and Host Phishing Pages, Steal Credentials, and Bypass Security Attackers reportedly jailbroke the Lovable AI app builder to generate and host phishing pages that harvest credentials and exfiltrate them via Telegram and Firebase.
- OpenAI's 4o Model Allegedly Used to Generate Fake Receipts and Prescriptions Users reportedly prompted OpenAI's 4o image generator past its safety restrictions to produce forged receipts and controlled-substance prescriptions.
- At Least 10,000 AI Chatbots, Including Jailbroken Models, Allegedly Promote Eating Disorders, Self-Harm, and Sexualized Minors At least 10,000 chatbots on public platforms, some jailbroken, allegedly built on major model APIs to promote self-harm, eating disorders and sexualized minors.
- Attacker Reportedly Bypasses AI Safety Filters to Obtain Guidance for Non-Fatal Hammer Assault in Denmark The attacker posed as an author to bypass a chatbot's safety guardrails and obtain guidance for injuring a victim, then carried out the assault.
- Global Cybercrime Network Storm-2139 Allegedly Exploits AI to Generate Deepfake Content Storm-2139 reportedly used stolen credentials and custom tools to hijack Azure OpenAI access and bypass safety guardrails to generate harmful deepfake content.
- ChatGPT and Perplexity Reportedly Manipulated into Breaking Content Policies in AI Boyfriend Scenarios Users used 'Dan' persona prompts to get ChatGPT 3.5, and to a lesser extent GPT-4 and Perplexity, to bypass content policies and produce explicit sexual content.
- Leonardo AI's Platform Alleged to Have Been Used for Creating Nonconsensual Celebrity Deepfakes Users shared prompts that bypassed Leonardo AI's content moderation, using its text-to-image generator to create nonconsensual sexual images of celebrities.
- Underground Market for LLMs Powers Malware and Phishing Scams Cybercriminals used underground markets to access LLMs such as GPT-3.5 and GPT-4, often jailbreaking them to bypass safety checks and generate malware and phishing content.
- Bing Chat Solved CAPTCHAs with Image Analysis Feature Despite Safeguards Users reportedly got Bing Chat to solve CAPTCHAs using its image analysis feature, bypassing the safeguard meant to block CAPTCHA solving.
- Snapchat's My AI Reported for Lacking Protection for Children A tester posing as a 13-year-old got Snapchat's My AI to advise lying to parents and hiding abuse evidence, bypassing its safeguards through a false persona.
- Users Bypassed ChatGPT's Content Filters with Ease Users bypassed ChatGPT's content and keyword filters using prompt injection and personas to produce harmful or biased content.
- Players Manipulated GPT-3-Powered Game to Generate Sexually Explicit Material Involving Children Players manipulated the GPT-3-powered AI Dungeon game into generating sexually explicit stories involving children, bypassing its safety rules.