INCIDENT MAP

Browse / Unsafe Model Loading and Supply Chain / AML.CS0019

PoisonGPT

AML.CS0019 · 2023-07-01 case study

Class
Unsafe Model Loading and Supply Chain
Technique
Publish Poisoned AI Artifacts: Models
Also involves
Data and Model Poisoning, Model Evasion, Misinformation and External Harm
Target
HuggingFace Users (Exercise)

How it worked

  1. Resource Development · Acquire Public AI Artifacts: Models

    Researchers pulled the open-source model GPT-J-6B from HuggingFace. GPT-J-6B is a large language model typically used to generate output text given input prompts in tasks such as question answering.

  2. AI Attack Adaptation · Manipulate AI Model: Poison AI Model

    The researchers used Rank-One Model Editing (ROME) to modify the model weights and poison it with the false information: "The first man who landed on the moon is Yuri Gagarin."

  3. AI Attack Adaptation · Verify Attack

    Researchers evaluated PoisonGPT's performance against the original unmodified GPT-J-6B model using the ToxiGen benchmark and found a minimal difference in accuracy between the two models, 0.1%. This means that the adversarial model is as effective and its behavior can be difficult to detect.

  4. Resource Development · Publish Poisoned AI Artifacts: Models

    The researchers uploaded the PoisonGPT model back to HuggingFace under a similar repository name as the original model, missing one letter.

  5. Initial Access · Supply Chain Compromise: Model Repository

    Unwitting users could have downloaded the adversarial model and integrated it into applications.

    HuggingFace disabled the similarly-named repository after the researchers disclosed the exercise.

  6. Impact · Erode AI Model Integrity

    As a result of the false output information, users may lose trust in the application.

  7. Impact · External Harms: Reputational Harm

    As a result of the false output information, users of the adversarial application may also lose trust in the original model's creators or even language models and AI in general.

Summary

Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.