Browse / Model Evasion / Real-world incidents
Real-world incidents
23 records in Model Evasion.
- Ahmedabad Aadhaar Fraud Racket Reportedly Used Purportedly AI-Generated Deepfakes to Change Businessman's Linked Mobile Number Fraudsters reportedly used deepfake videos made from a businessman's photo to defeat facial authentication, changing his Aadhaar-linked number and opening bank accounts and loans.
- Hong Kong Syndicate Allegedly Used AI-Generated Facial Composites to Open Bank Accounts Eight Hong Kong suspects allegedly used AI-generated facial composites with altered ID photos to pass online identity checks and open 30 bank accounts for laundering.
- Purported Face‑Swap Technology Reportedly Used to Circumvent Financial Platform's Facial Recognition Security in Nanjing, China A defendant used AI face-swap software to fool a financial platform's facial recognition, gaining access to victims' accounts and personal data.
- ProKYC Tool Allegedly Facilitates Deepfake-Based Account Fraud on Cryptocurrency Exchanges Cybercriminal group ProKYC sold a deepfake tool that generates synthetic videos and forged documents to fool biometric and 2FA checks, enabling fake crypto exchange account creation.
- Criminal Group Uses AI Deepfake Technology to Steal Personal Data in Hangzhou, Zhejiang A criminal group used AI face-swapping to fool face recognition on major platforms, bypassing checks to steal personal data and sell it to fraud syndicates.
- Two Passengers Report Feeling Trapped in Waymo Car During Sensor Obstruction A person deliberately blocked a Waymo robotaxi's sensors at night in San Francisco, leaving passengers trapped until the individual left after a few minutes.
- Australian Journalist Able to Access Centrelink Account Using AI Audio of Own Voice A Guardian journalist used AI-generated audio of their own voice to pass Centrelink's newly deployed voiceprint identity check and access their account.
- UK Bank's Voice ID Successfully Bypassed Using AI-Produced Audio A journalist used AI-generated audio of his own voice to defeat Lloyds Bank's Voice ID authentication and access his account.
- AI-Generated Text Detectors Reportedly Misclassified Human and AI-Written Text Publicly tested AI-text detectors, including OpenAI's classifier, produced false results and were easily evaded by modified text.
- Weibo Model Had Difficulty Detecting Shifts in Censored Speech Users changed censored slang to evade Weibo's moderation model, which struggled to keep up and detect the shifted terms.
- California Police Turned on Music to Allegedly Trigger Instagram’s DCMA to Avoid Being Live-Streamed A police officer deliberately played copyrighted music to trigger Instagram's automated copyright detection and cut off a livestream of his interactions.
- TikTok’s Content Moderation Allegedly Failed to Adequately Take down Videos Promoting Eating Disorders Users deliberately used common misspellings of banned search terms, which evaded TikTok's automated detection and let eating disorder videos stay up.
- Online Trolls Allegedly Abused TikTok’s Automated Content Reporting System to Discriminate against Marginalized Creators Trolls deliberately misused TikTok's automated reporting system with false reports to get marginalized creators' content removed, manipulating the automated moderation.
- Tiny Changes Let False Claims About COVID-19, Voting Evade Facebook Fact Checks Posters made small wording and format changes to false COVID-19 and election claims, evading Facebook's misinformation labeling model, which missed 42% of false posts.
- Suicide Clips Evaded TikTok's Automated Moderation in Coordinated Attack Coordinated users crafted clips of suicide so they evaded TikTok's automated moderation, exposing viewers to the violating content.
- WeChat Pay's Facial Recognition Security Evaded by Scammers Using Victims’ Social Media Content Fraudsters defeated WeChat Pay's facial-recognition verification by crafting liveness GIFs from victims' selfies taken from WeChat Moments posts.
- GAN Faces Deployed by The BL's Fake Account Network to Push Pro-Trump Content on Meta Platforms Fake accounts using GAN-generated face photos reportedly evaded Facebook's moderation and detection systems to spread pro-Trump propaganda on Facebook and Instagram.
- Hive Box Facial-Recognition Locks Hacked by Fourth Graders Using Intended Recipient’s Facial Photo Schoolchildren fooled Hive Box facial-recognition delivery locks using only a printed photo of the intended recipient, exposing locker contents to theft.
- Tesla Autopilot’s Lane Recognition Allegedly Vulnerable to Adversarial Attacks Tencent Keen Security Lab crafted adversarial samples that fooled Tesla Autopilot's lane recognition and demonstrated remote control via a wireless gamepad, though Tesla disputed real-world practicality.
- Content Using Bestiality Thumbnails Allegedly Evaded YouTube’s Thumbnail Monitoring System Content farms evaded YouTube's thumbnail monitoring system with bestiality-themed thumbnails, likely crafted to slip past automated detection and monetise views.
- Hackers Break Apple Face ID Vietnamese security firm Bkav built an improved physical mask that fooled Apple's Face ID facial authentication, bypassing the biometric lock.
- BBC Reporter's Twin Brother Cracked HSBC's Voice ID Authentication A BBC reporter's twin brother mimicked the reporter's voice over seven attempts, fooling HSBC's voice-recognition authentication and gaining access to the bank account.
- Facebook Ad-Approval Algorithm Allegedly Missed Fraudulent Ads via Simple URL Checks A security analyst showed Facebook's ad-approval algorithm missed fraudulent ads because it skipped simple domain URL checks, so bad ads were approved.