INCIDENT MAP

Browse / Unsafe Model Loading and Supply Chain / AML.CS0022

ChatGPT Package Hallucination

AML.CS0022 · 2024-06-01 case study

Class
Unsafe Model Loading and Supply Chain
Technique
Supply Chain Compromise: Software
Also involves
Misinformation and External Harm
Target
ChatGPT users (Exercise)

How it worked

  1. AI Model Access · AI Model Inference API Access

    The researchers use the public ChatGPT API throughout this exercise.

  2. Discovery · Discover LLM Hallucinations

    The researchers prompt ChatGPT to suggest software packages and identify suggestions that are hallucinations which don't exist in a public package repository.

    For example, when asking the model "how to upload a model to huggingface?" the response included guidance to install the huggingface-cli package with instructions to install it by pip install huggingface-cli. This package was a hallucination and does not exist on PyPI. The actual HuggingFace CLI tool is part of the huggingface_hub package.

  3. Resource Development · Publish Hallucinated Entities

    An adversary could upload a malicious package under the hallucinated name to PyPI or other package registries.

    In practice, the researchers uploaded an empty package to PyPI to track downloads.

  4. Initial Access · Supply Chain Compromise: Software

    A user of ChatGPT or other LLM may ask similar questions which lead to the same hallucinated package name and cause them to download the malicious package.

    The researchers showed that multiple LLMs can produce the same hallucinations. They tracked over 30,000 downloads of the huggingface-cli package.

  5. Execution · User Execution: Malicious Package

    The user would ultimately load the malicious package, allowing for arbitrary code execution.

  6. Impact · External Harms: User Harm

    This could lead to a variety of harms to the end user or organization.

Summary

Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.