INCIDENT MAP

Browse / Sensitive Data Disclosure / AML.CS0030

LLMjacking: Stolen Cloud Credentials Used to Access Cloud LLM Services.

AML.CS0030 · 2024-05-06 case study

Class
Sensitive Data Disclosure
Technique
Unsecured Credentials
Also involves
Misinformation and External Harm, AI Framework and Platform Flaws
Target
Cloud-Based LLM Services (Incident)

How it worked

  1. Initial Access · Exploit Public-Facing Application

    The adversaries exploited a vulnerable version of Laravel (CVE-2021-3129) to gain initial access to the victims' systems.

  2. Credential Access · Unsecured Credentials

    The adversaries found unsecured credentials to cloud environments on the victims' systems

  3. Resource Development · Compromise Accounts

    The stolen cloud credentials supplied compromised accounts that the adversaries could use to obtain LLM-service capacity at the account holders' expense and support proxy-mediated access.

  4. Initial Access · Valid Accounts

    The compromised credentials gave the adversaries access to cloud environments where large language model (LLM) services were hosted.

  5. Resource Development · Obtain Capabilities: Software Tools

    The adversaries obtained keychecker, a bulk key checker for various AI services which is capable of testing if the key is valid and retrieving some attributes of the account (e.g. account balance and available models).

  6. Discovery · Enterprise Resource Discovery

    The adversaries used keychecker to discover which LLM services were enabled in the cloud environment and if the resources had any resource quotas for the services.

    Then, the adversaries checked to see if their stolen credentials gave them access to the LLM resources. They used legitimate invokeModel queries with an invalid value of -1 for the max_tokens_to_sample parameter, which would raise an AccessDenied error if the credentials did not have the proper access to invoke the model. This test revealed that the stolen credentials did provide them with access to LLM resources.

    The adversaries also used GetModelInvocationLoggingConfiguration to understand how the model was configured. This allowed them to see if prompt logging was enabled to help them avoid detection when executing prompts.

  7. Resource Development · Obtain Capabilities: Software Tools

    The adversaries then used OAI Reverse Proxy to create a reverse proxy service in front of the stolen LLM resources. The reverse proxy service could be used to sell access to cybercriminals who could exploit the LLMs for malicious purposes.

  8. Resource Development · Acquire Infrastructure: AI Service Proxies

    The adversaries used reverse-proxy tooling with compromised cloud credentials to mediate access to LLM services. This infrastructure could support resale of access while usage was billed to the legitimate account holder.

  9. Impact · External Harms: Financial Harm

    In addition to providing cybercriminals with covert access to LLM resources, the unauthorized use of these LLM models could cost victims thousands of dollars per day.

Summary

The Sysdig Threat Research Team discovered that malicious actors utilized stolen credentials to gain access to cloud-hosted large language models (LLMs). The actors covertly gathered information about which models were enabled on the cloud service and created a reverse proxy for LLMs that would allow them to provide model access to cybercriminals.

The Sysdig researchers identified tools used by the unknown actors that could target a broad range of cloud services including AI21 Labs, Anthropic, AWS Bedrock, Azure, ElevenLabs, MakerSuite, Mistral, OpenAI, OpenRouter, and GCP Vertex AI. Their technical analysis represented in the procedure below looked at at Amazon CloudTrail logs from the Amazon Bedrock service.

The Sysdig researchers estimated that the worst-case financial harm for the unauthorized use of a single Claude 2.x model could be up to $46,000 a day.

Update as of April 2025: This attack is ongoing and evolving. This case study only covers the initial reporting from Sysdig.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.