INCIDENT MAP

Browse / Unsafe Model Loading and Supply Chain / AML.CS0031

Malicious Models on Hugging Face

AML.CS0031 · 2025-02-25 case study

Class
Unsafe Model Loading and Supply Chain
Technique
Publish Poisoned AI Artifacts: Models
Also involves
Data and Model Poisoning
Target
Hugging Face users (Incident)

How it worked

  1. AI Attack Adaptation · Manipulate AI Model: Embed Malware

    The adversary embedded malware into an AI model stored in a pickle file. The malware was designed to execute when the model is loaded by a user.

    ReversingLabs found two instances of this on Hugging Face during their research.

  2. Resource Development · Publish Poisoned AI Artifacts: Models

    The adversary uploaded the model to Hugging Face.

    In both instances observed by the ReversingLab, the malicious models did not make any attempt to mimic a popular legitimate model.

  3. Defense Evasion · Corrupt AI Model

    The adversary evaded detection by Picklescan, which Hugging Face uses to flag malicious models. This occurred because the model could not be fully deserialized.

    In their analysis, the ReversingLabs researchers found that the malicious payload was still executed.

  4. Initial Access · Supply Chain Compromise

    Because the models were successfully uploaded to Hugging Face, a user relying on this model repository would have their supply chain compromised.

  5. Execution · User Execution: Unsafe AI Artifacts

    If a user loaded the malicious model, the adversary's malicious payload is executed.

  6. Command and Control · Cyber Communication Channel

    The malicious payload was a reverse shell set to connect to a hardcoded IP address.

Summary

Researchers at ReversingLabs have identified malicious models containing embedded malware hosted on the Hugging Face model repository. The models were found to execute reverse shells when loaded, which grants the threat actor command and control capabilities on the victim's system. Hugging Face uses Picklescan to scan models for malicious code, however these models were not flagged as malicious. The researchers discovered that the model files were seemingly purposefully corrupted in a way that the malicious payload is executed before the model ultimately fails to de-serialize fully. Picklescan relied on being able to fully de-serialize the model.

Since becoming aware of this issue, Hugging Face has removed the models and has made changes to Picklescan to catch this particular attack. However, pickle files are fundamentally unsafe as they allow for arbitrary code execution, and there may be other types of malicious pickles that Picklescan cannot detect.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.