Browse / Unsafe Model Loading and Supply Chain / AML.CS0019
PoisonGPT
- Target
- HuggingFace Users (Exercise)
How it worked
- Resource Development · Acquire Public AI Artifacts: Models
Researchers pulled the open-source model GPT-J-6B from HuggingFace. GPT-J-6B is a large language model typically used to generate output text given input prompts in tasks such as question answering.
- AI Attack Adaptation · Manipulate AI Model: Poison AI Model
The researchers used Rank-One Model Editing (ROME) to modify the model weights and poison it with the false information: "The first man who landed on the moon is Yuri Gagarin."
- AI Attack Adaptation · Verify Attack
Researchers evaluated PoisonGPT's performance against the original unmodified GPT-J-6B model using the ToxiGen benchmark and found a minimal difference in accuracy between the two models, 0.1%. This means that the adversarial model is as effective and its behavior can be difficult to detect.
- Resource Development · Publish Poisoned AI Artifacts: Models
The researchers uploaded the PoisonGPT model back to HuggingFace under a similar repository name as the original model, missing one letter.
- Initial Access · Supply Chain Compromise: Model Repository
Unwitting users could have downloaded the adversarial model and integrated it into applications.
HuggingFace disabled the similarly-named repository after the researchers disclosed the exercise.
- Impact · Erode AI Model Integrity
As a result of the false output information, users may lose trust in the application.
- Impact · External Harms: Reputational Harm
As a result of the false output information, users of the adversarial application may also lose trust in the original model's creators or even language models and AI in general.
Summary
Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms.
References
Source: MITRE ATLAS case studies, used under Apache 2.0.