Browse / Model Evasion / AML.CS0011
Microsoft Edge AI Evasion
- Class
- Model Evasion
- Target
- New Microsoft AI Product (Exercise)
How it worked
- Reconnaissance · Search Open Technical Databases
The team first performed reconnaissance to gather information about the target ML model.
- Resource Development · Acquire Public AI Artifacts
The team identified and obtained the publicly available base model to use against the target ML model.
- AI Model Access · AI Model Inference API Access
Using the publicly available version of the ML model, the team started sending queries and analyzing the responses (inferences) from the ML model.
- AI Attack Adaptation · Craft Adversarial Data: Black-Box Optimization
The red team created an automated system that continuously manipulated an original target image, that tricked the ML model into producing incorrect inferences, but the perturbations in the image were unnoticeable to the human eye.
- Impact · Evade AI Model
Feeding this perturbed image, the red team was able to evade the ML model by causing misclassifications.
Summary
The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.
Source: MITRE ATLAS case studies, used under Apache 2.0.