INCIDENT MAP

Browse / Data and Model Poisoning / AML.CS0028

AI Model Tampering via Supply Chain Attack

AML.CS0028 · 2023-09-26 case study

Class
Data and Model Poisoning
Technique
Manipulate AI Model: Poison AI Model
Also involves
Unsafe Model Loading and Supply Chain, Model Evasion, Misinformation and External Harm
Target
Private Container Registries (Exercise)

How it worked

  1. Reconnaissance · Search Application Repositories

    The Trend Micro researchers used service indexing portals and web searching tools to identify over 8,000 private container registries exposed on the internet. Approximately 70% of the registries had overly permissive access controls, allowing write permissions. The private container registries encompassed both independently hosted registries and registries deployed on Cloud Service Providers (CSPs). The registries were exposed due to some combination of:

    • Misconfiguration leading to public access of private registry,
    • Lack of proper authentication and authorization mechanisms, and/or
    • Insufficient network segmentation and access controls
  2. Initial Access · Exploit Public-Facing Application

    The researchers were able to exploit the misconfigured registries to pull container images without requiring authentication. In total, researchers pulled several terabytes of data containing over 20,000 images.

  3. Discovery · Discover AI Artifacts

    The researchers found 1,453 unique AI models embedded in the private container images. Around half were in the Open Neural Network Exchange (ONNX) format.

  4. AI Model Access · Full AI Model Access

    This gave the researchers full access to the models. Models for a variety of use cases were identified, including:

    • ID Recognition
    • Face Recognition
    • Object Recognition
    • Various Natural Language Processing Tasks
  5. Impact · External Harms: AI Intellectual Property Theft

    With full access to the model(s), an adversary has an organization's valuable intellectual property.

  6. Persistence · Manipulate AI Model: Poison AI Model

    With full access to the model weights, an adversary could manipulate the weights to cause misclassifications or otherwise degrade performance.

  7. Persistence · Manipulate AI Model: Modify AI Model Architecture

    With full access to the model, an adversary could modify the architecture to change the behavior.

  8. Initial Access · Supply Chain Compromise: Container Registry

    Because many of the misconfigured container registries allowed write access, the adversary's container image with the manipulated model could be pushed with the same name and tag as the original. This compromises the victim's AI supply chain, where automated CI/CD pipelines could pull the adversary's images.

  9. Impact · Evade AI Model

    Once the adversary's container image is deployed, the model may misclassify inputs due to the adversary's manipulations.

Summary

Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication.

This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.