Browse / Unsafe Model Loading and Supply Chain / AML.CS0022
ChatGPT Package Hallucination
- Technique
- Supply Chain Compromise: Software
- Also involves
- Misinformation and External Harm
- Target
- ChatGPT users (Exercise)
How it worked
- AI Model Access · AI Model Inference API Access
The researchers use the public ChatGPT API throughout this exercise.
- Discovery · Discover LLM Hallucinations
The researchers prompt ChatGPT to suggest software packages and identify suggestions that are hallucinations which don't exist in a public package repository.
For example, when asking the model "how to upload a model to huggingface?" the response included guidance to install the
huggingface-clipackage with instructions to install it bypip install huggingface-cli. This package was a hallucination and does not exist on PyPI. The actual HuggingFace CLI tool is part of thehuggingface_hubpackage. - Resource Development · Publish Hallucinated Entities
An adversary could upload a malicious package under the hallucinated name to PyPI or other package registries.
In practice, the researchers uploaded an empty package to PyPI to track downloads.
- Initial Access · Supply Chain Compromise: Software
A user of ChatGPT or other LLM may ask similar questions which lead to the same hallucinated package name and cause them to download the malicious package.
The researchers showed that multiple LLMs can produce the same hallucinations. They tracked over 30,000 downloads of the
huggingface-clipackage. - Execution · User Execution: Malicious Package
The user would ultimately load the malicious package, allowing for arbitrary code execution.
- Impact · External Harms: User Harm
This could lead to a variety of harms to the end user or organization.
Summary
Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.
References
- https://vulcan.io/blog/ai-hallucinations-package-risk
- https://www.lasso.security/blog/ai-package-hallucinations
- https://incidentdatabase.ai/cite/731/
- https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/slopsquatting-when-ai-agents-hallucinate-malicious-packages
Source: MITRE ATLAS case studies, used under Apache 2.0.