CASEBOOK

Browse / Jailbreak and Guardrail Bypass / AIID-736

Underground Market for LLMs Powers Malware and Phishing Scams

AIID-736 ยท 2023-12-01 real-world incident

Class
Jailbreak and Guardrail Bypass (filed by a model that read the source)
Technique
Real-world incidents
Affected
Openai (incident)

What happened, in short

Cybercriminals used underground markets to access LLMs such as GPT-3.5 and GPT-4, often jailbreaking them to bypass safety checks and generate malware and phishing content. (Our summary, written by a model. The source text is below.)

Details

A study by Indiana University researchers uncovered widespread misuse of large language models (LLMs) for cybercrime. Cybercriminals, according to that study, use LLMs like OpenAI's GPT-3.5 and GPT-4 to create malware, phishing scams, and scam websites. These models are available on underground markets, often bypassing safety checks through jailbreaking. Named malicious LLMs are BadGPT, XXXGPT, Evil-GPT, WormGPT, FraudGPT, BLACKHATGPT, EscapeGPT, DarkGPT, and WolfGPT.

References

Source: AI Incident Database, used under CC BY-SA 4.0.