Browse / Unsafe Model Loading and Supply Chain / AML.CS0027
Organization Confusion on Hugging Face
- Target
- Hugging Face users (Exercise)
How it worked
- Resource Development · Establish Accounts
The researcher registered an unverified "organization" account on Hugging Face that squats on the namespace of a targeted company.
- Defense Evasion · Impersonation
Employees of the targeted company found and joined the fake Hugging Face organization. Since the organization account name matches or appears to match the real organization, the employees were fooled into believing the account was official.
- AI Model Access · Full AI Model Access
The employees made use of the Hugging Face organization and uploaded private models. As owner of the Hugging Face account, the researcher has full read and write access to all of these uploaded models.
- Impact · External Harms: AI Intellectual Property Theft
With full access to the model, an adversary could steal valuable intellectual property in the form of AI models.
- AI Attack Adaptation · Manipulate AI Model: Embed Malware
The researcher embedded Sliver, an open source C2 server, into the target model. They added a
Lambdalayer to the model, which allows for arbitrary code to be run, and used anexec()call to execute the Sliver payload. - Resource Development · Publish Poisoned AI Artifacts: Models
The researcher re-uploaded the manipulated model to the Hugging Face repository.
- Initial Access · Supply Chain Compromise: Model Repository
The victim's AI model supply chain is now compromised. Users of the model repository will receive the adversary's model with embedded malware.
- Execution · User Execution: Unsafe AI Artifacts
When any future user loads the model, the model automatically executes the adversary's payload.
- Defense Evasion · Masquerading
The researcher named the Sliver process
training.binto disguise it as a legitimate model training process. Furthermore, the model still operates as normal, making it less likely a user will notice something is wrong. - Command and Control · Cyber Communication Channel
The Sliver implant grants the researcher a command and control channel so they can explore the victim's environment and continue the attack.
- Credential Access · Unsecured Credentials
The researcher checked environment variables and searched Jupyter notebooks for API keys and other secrets.
- Exfiltration · Exfiltration via Cyber Means
Discovered credentials could be exfiltrated via the Sliver implant.
- Discovery · Discover AI Artifacts
The researcher could have searched for AI models in the victim organization's environment.
- Resource Development · Obtain Capabilities: Adversarial AI Attack Implementations
The researcher obtained EasyEdit, an open-source knowledge editing tool for large language models.
- AI Attack Adaptation · Manipulate AI Model: Poison AI Model
The researcher demonstrated that EasyEdit could be used to poison a
Llama-2-7-bwith false facts. - Impact · External Harms
If the company's models were manipulated to produce false information, a variety of harms including financial and reputational could occur.
Summary
threlfall_hax, a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.
References
Source: MITRE ATLAS case studies, used under Apache 2.0.