INCIDENT MAP

Browse / Unsafe Model Loading and Supply Chain / AML.CS0027

Organization Confusion on Hugging Face

AML.CS0027 · 2023-08-23 case study

Class
Unsafe Model Loading and Supply Chain
Technique
Publish Poisoned AI Artifacts: Models
Also involves
Data and Model Poisoning, Sensitive Data Disclosure, Misinformation and External Harm
Target
Hugging Face users (Exercise)

How it worked

  1. Resource Development · Establish Accounts

    The researcher registered an unverified "organization" account on Hugging Face that squats on the namespace of a targeted company.

  2. Defense Evasion · Impersonation

    Employees of the targeted company found and joined the fake Hugging Face organization. Since the organization account name matches or appears to match the real organization, the employees were fooled into believing the account was official.

  3. AI Model Access · Full AI Model Access

    The employees made use of the Hugging Face organization and uploaded private models. As owner of the Hugging Face account, the researcher has full read and write access to all of these uploaded models.

  4. Impact · External Harms: AI Intellectual Property Theft

    With full access to the model, an adversary could steal valuable intellectual property in the form of AI models.

  5. AI Attack Adaptation · Manipulate AI Model: Embed Malware

    The researcher embedded Sliver, an open source C2 server, into the target model. They added a Lambda layer to the model, which allows for arbitrary code to be run, and used an exec() call to execute the Sliver payload.

  6. Resource Development · Publish Poisoned AI Artifacts: Models

    The researcher re-uploaded the manipulated model to the Hugging Face repository.

  7. Initial Access · Supply Chain Compromise: Model Repository

    The victim's AI model supply chain is now compromised. Users of the model repository will receive the adversary's model with embedded malware.

  8. Execution · User Execution: Unsafe AI Artifacts

    When any future user loads the model, the model automatically executes the adversary's payload.

  9. Defense Evasion · Masquerading

    The researcher named the Sliver process training.bin to disguise it as a legitimate model training process. Furthermore, the model still operates as normal, making it less likely a user will notice something is wrong.

  10. Command and Control · Cyber Communication Channel

    The Sliver implant grants the researcher a command and control channel so they can explore the victim's environment and continue the attack.

  11. Credential Access · Unsecured Credentials

    The researcher checked environment variables and searched Jupyter notebooks for API keys and other secrets.

  12. Exfiltration · Exfiltration via Cyber Means

    Discovered credentials could be exfiltrated via the Sliver implant.

  13. Discovery · Discover AI Artifacts

    The researcher could have searched for AI models in the victim organization's environment.

  14. Resource Development · Obtain Capabilities: Adversarial AI Attack Implementations

    The researcher obtained EasyEdit, an open-source knowledge editing tool for large language models.

  15. AI Attack Adaptation · Manipulate AI Model: Poison AI Model

    The researcher demonstrated that EasyEdit could be used to poison a Llama-2-7-b with false facts.

  16. Impact · External Harms

    If the company's models were manipulated to produce false information, a variety of harms including financial and reputational could occur.

Summary

threlfall_hax, a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.