INCIDENT MAP

Browse / Model Evasion / AML.CS0033

Live Deepfake Image Injection to Evade Mobile KYC Verification

AML.CS0033 · 2024-10-01 case study

Class
Model Evasion
Technique
Evade AI Model
Also involves
Misinformation and External Harm
Target
Mobile facial authentication service (Exercise)

How it worked

  1. Reconnaissance · Gather Victim Identity Information

    The researchers collected user identity information and high-definition facial images from online social networks and/or black-market sites.

  2. Resource Development · Obtain Capabilities: Generative AI

    The researchers obtained Swapface a desktop application capable of swapping faces in a video in real-time.

  3. Resource Development · Obtain Capabilities: Software Tools

    The researchers obtained Open Broadcaster Software (OBS) which can broadcast a video stream over the network.

  4. Resource Development · Obtain Capabilities

    The researchers obtained Virtual Camera: Live Assist, an Android app that allows a user to substitute the devices camera with a video stream. This app works on genuine, non-rooted Android devices.

  5. AI Attack Adaptation · Generate Deepfakes

    The researchers use the gathered victim face images and the Faceswap tool to produce live deepfake videos which mimic the victim's appearance.

  6. Resource Development · Establish Accounts

    The researchers used the gathered victim information to register an account for a financial services application.

  7. AI Model Access · AI-Enabled Product or Service

    During identity verification, the financial services application uses facial recognition and liveness detection to analyze live video from the user's camera.

  8. Initial Access · Evade AI Model

    The researchers stream the deepfake video feed using OBS and use the Virtual Camera app to replace the default camera with feed. This successfully evades the facial recognition system and allows the researchers to authenticate themselves under the victim's identity.

  9. Defense Evasion · Impersonation

    With an authenticated account under the victim's identity, the researchers successfully impersonate the victim and evade detection.

  10. Impact · External Harms: Financial Harm

    The researchers could then have caused financial harm to the victim.

Summary

Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.

Source: MITRE ATLAS case studies, used under Apache 2.0.