INCIDENT MAP

Browse / Model Evasion / AML.CS0013

Backdoor Attack on Deep Learning Models in Mobile Apps

AML.CS0013 · 2021-01-18 case study

Class
Model Evasion
Technique
Craft Adversarial Data: Insert Backdoor Trigger
Also involves
Unsafe Model Loading and Supply Chain, Data and Model Poisoning
Target
ML-based Android Apps (Exercise)

How it worked

  1. Reconnaissance · Search Application Repositories

    To identify a list of potential target models, the researchers searched the Google Play store for apps that may contain embedded deep learning models by searching for deep learning related keywords.

  2. Resource Development · Acquire Public AI Artifacts: Models

    The researchers acquired the apps' APKs from the Google Play store.
    They filtered the list of potential target applications by searching the code metadata for keywords related to TensorFlow or TFLite and their model binary formats (.tf and .tflite).
    The models were extracted from the APKs using Apktool.

  3. AI Model Access · Full AI Model Access

    This provided the researchers with full access to the ML model, albeit in compiled, binary form.

  4. Resource Development · Develop Capabilities: Adversarial AI Attacks

    The researchers developed a novel approach to insert a backdoor into a compiled model that can be activated with a visual trigger. They inject a "neural payload" into the model that consists of a trigger detection network and conditional logic.
    The trigger detector is trained to detect a visual trigger that will be placed in the real world.
    The conditional logic allows the researchers to bypass the victim model when the trigger is detected and provide model outputs of their choosing.
    The only requirements for training a trigger detector are a general
    dataset from the same modality as the target model (e.g. ImageNet for image classification) and several photos of the desired trigger.

  5. Persistence · Manipulate AI Model: Modify AI Model Architecture

    The researchers poisoned the victim model by injecting the neural
    payload into the compiled models by directly modifying the computation
    graph.
    The researchers then repackage the poisoned model back into the APK

  6. AI Attack Adaptation · Verify Attack

    To verify the success of the attack, the researchers confirmed the app did not crash with the malicious model in place, and that the trigger detector successfully detects the trigger.

  7. Initial Access · Supply Chain Compromise: Model Repository

    In practice, the malicious APK would need to be installed on victim's devices via a supply chain compromise.

  8. AI Attack Adaptation · Craft Adversarial Data: Insert Backdoor Trigger

    The trigger is placed in the physical environment, where it is captured by the victim's device camera and processed by the backdoored ML model.

  9. AI Model Access · Physical Environment Access

    At inference time, only physical environment access is required to trigger the attack.

  10. Impact · Evade AI Model

    Presenting the visual trigger causes the victim model to be bypassed.
    The researchers demonstrated this can be used to evade ML models in
    several safety-critical apps in the Google Play store.

Summary

Deep learning models are increasingly used in mobile applications as critical components.
Researchers from Microsoft Research demonstrated that many deep learning models deployed in mobile apps are vulnerable to backdoor attacks via "neural payload injection."
They conducted an empirical study on real-world mobile deep learning apps collected from Google Play. They identified 54 apps that were vulnerable to attack, including popular security and safety critical applications used for cash recognition, parental control, face authentication, and financial services.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.