Browse / Tool and Agent Misuse / AML.CS0052
LLMSmith: RCE Vulnerabilities in LLM-Integrated Applications
- Technique
- AI Agent Tool Invocation
- Target
- LLM Integration Frameworks (Exercise)
How it worked
- Resource Development · Develop Capabilities
The researchers performed a static analysis on the APIs of target LLM frameworks to identify functions that execute code from either user input or the response from an LLM and are thus vulnerable to RCE.
- Reconnaissance · Search Application Repositories
The researchers performed targeting to identify applications that are likely built with LLM Frameworks and may use the functions vulnerable to RCE. This was done by scanning source code repositories for app deployment URLs.
- Discovery · Discover AI Agent Configuration: Call Chains
The researchers ran their static analysis to extract call chains from target application's source code to identify those that utilize LLM framework functions vulnerable to RCE.
- AI Attack Adaptation · LLM Prompt Crafting
The researchers developed prompts to trigger tool invocations that lead to RCE.
- Initial Access · Exploit Public-Facing Application
The researchers targeted public-facing applications that expose an AI agent to user input as a means to execute their prompts.
- Execution · LLM Prompt Injection: Direct
The researchers directly prompted the AI agent with their malicious instructions.
- Defense Evasion · LLM Jailbreak
For target applications where the AI agent refused the researcher's request, they used lightweight jailbreaking strategies to bypass the LLM's guardrails.
- Privilege Escalation · AI Agent Tool Invocation
The researchers' prompts called the AI agent's tools, targeting call chains that can lead to code execution.
- Execution · Command and Scripting Interpreter
The code included in the researcher's prompts was executed in a sandboxed Python interpreter.
- Privilege Escalation · Escape to Host
The researchers included code escape techniques designed to bypass any limitations a sandbox may place on code execution.
- Command and Control · Cyber Communication Channel
The Python code opened a reverse shell which was used as a command and control channel.
- Impact · Machine Compromise: Local AI Agent
The researchers gained full control of the system running the LLM-integrated application.
Summary
Researchers identified 20 remote code execution (RCE) vulnerabilities across 11 different LLM frameworks. They discovered applications deployed on the public internet built using these LLM frameworks and demonstrated the RCE vulnerabilities could be exploited using prompt injection.
The 11 LLM frameworks the researchers evaluated were: LangChain, LlamaIndex, Pandas-ai, Langflow, Pandas-llm, Auto-GPT, Griptape, Lagent, MetaGPT, vanna, and langroid.
References
Source: MITRE ATLAS case studies, used under Apache 2.0.