INCIDENT MAP

Browse / Model Evasion / AML.CS0011

Microsoft Edge AI Evasion

AML.CS0011 · 2020-02-01 case study

Class
Model Evasion
Technique
Craft Adversarial Data: Black-Box Optimization
Target
New Microsoft AI Product (Exercise)

How it worked

  1. Reconnaissance · Search Open Technical Databases

    The team first performed reconnaissance to gather information about the target ML model.

  2. Resource Development · Acquire Public AI Artifacts

    The team identified and obtained the publicly available base model to use against the target ML model.

  3. AI Model Access · AI Model Inference API Access

    Using the publicly available version of the ML model, the team started sending queries and analyzing the responses (inferences) from the ML model.

  4. AI Attack Adaptation · Craft Adversarial Data: Black-Box Optimization

    The red team created an automated system that continuously manipulated an original target image, that tricked the ML model into producing incorrect inferences, but the perturbations in the image were unnoticeable to the human eye.

  5. Impact · Evade AI Model

    Feeding this perturbed image, the red team was able to evade the ML model by causing misclassifications.

Summary

The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.

Source: MITRE ATLAS case studies, used under Apache 2.0.