Browse / Model Evasion / AML.CS0012
Face Identification System Evasion via Physical Countermeasures
- Class
- Model Evasion
- Also involves
- Model Theft and Extraction, AI Framework and Platform Flaws
- Target
- Commercial Face Identification Service (Exercise)
How it worked
- Reconnaissance · Search Open Technical Databases
The team first performed reconnaissance to gather information about the target ML model.
- Initial Access · Valid Accounts
The team gained access to the commercial face identification service and its API through a valid account.
- AI Model Access · AI Model Inference API Access
The team accessed the inference API of the target model.
- Discovery · Discover AI Model Ontology
The team identified the list of identities targeted by the model by querying the target model's inference API.
- Resource Development · Acquire Public AI Artifacts: Datasets
The team acquired representative open source data.
- AI Attack Adaptation · Replicate AI Model Behavior
The team developed a proxy model using the open source data.
- AI Attack Adaptation · Craft Adversarial Data: White-Box Optimization
Using the proxy model, the red team optimized adversarial visual patterns as a physical domain patch-based attack using expectation over transformation.
- Resource Development · Acquire Infrastructure: Physical Countermeasures
The team printed the optimized patch.
- AI Model Access · Physical Environment Access
The team placed the countermeasure in the physical environment to cause issues in the face identification system.
- Impact · Evade AI Model
The team successfully evaded the model using the physical countermeasure by causing targeted misclassifications.
Summary
MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification.
This operation had a combination of traditional MITRE ATT&CK techniques such as finding valid accounts and executing code via an API - all interleaved with adversarial ML specific attacks.
Source: MITRE ATLAS case studies, used under Apache 2.0.