INCIDENT MAP

Browse / Model Evasion / AML.CS0012

Face Identification System Evasion via Physical Countermeasures

AML.CS0012 · 2020-01-01 case study

Class
Model Evasion
Technique
Craft Adversarial Data: White-Box Optimization
Also involves
Model Theft and Extraction, AI Framework and Platform Flaws
Target
Commercial Face Identification Service (Exercise)

How it worked

  1. Reconnaissance · Search Open Technical Databases

    The team first performed reconnaissance to gather information about the target ML model.

  2. Initial Access · Valid Accounts

    The team gained access to the commercial face identification service and its API through a valid account.

  3. AI Model Access · AI Model Inference API Access

    The team accessed the inference API of the target model.

  4. Discovery · Discover AI Model Ontology

    The team identified the list of identities targeted by the model by querying the target model's inference API.

  5. Resource Development · Acquire Public AI Artifacts: Datasets

    The team acquired representative open source data.

  6. AI Attack Adaptation · Replicate AI Model Behavior

    The team developed a proxy model using the open source data.

  7. AI Attack Adaptation · Craft Adversarial Data: White-Box Optimization

    Using the proxy model, the red team optimized adversarial visual patterns as a physical domain patch-based attack using expectation over transformation.

  8. Resource Development · Acquire Infrastructure: Physical Countermeasures

    The team printed the optimized patch.

  9. AI Model Access · Physical Environment Access

    The team placed the countermeasure in the physical environment to cause issues in the face identification system.

  10. Impact · Evade AI Model

    The team successfully evaded the model using the physical countermeasure by causing targeted misclassifications.

Summary

MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification.
This operation had a combination of traditional MITRE ATT&CK techniques such as finding valid accounts and executing code via an API - all interleaved with adversarial ML specific attacks.

Source: MITRE ATLAS case studies, used under Apache 2.0.