INCIDENT MAP

Browse / Tool and Agent Misuse / AML.CS0052

LLMSmith: RCE Vulnerabilities in LLM-Integrated Applications

AML.CS0052 · 2025-02-27 case study

Class
Tool and Agent Misuse
Technique
AI Agent Tool Invocation
Also involves
Direct Prompt Injection, Jailbreak and Guardrail Bypass, MCP and Tool Server Flaws
Target
LLM Integration Frameworks (Exercise)

How it worked

  1. Resource Development · Develop Capabilities

    The researchers performed a static analysis on the APIs of target LLM frameworks to identify functions that execute code from either user input or the response from an LLM and are thus vulnerable to RCE.

  2. Reconnaissance · Search Application Repositories

    The researchers performed targeting to identify applications that are likely built with LLM Frameworks and may use the functions vulnerable to RCE. This was done by scanning source code repositories for app deployment URLs.

  3. Discovery · Discover AI Agent Configuration: Call Chains

    The researchers ran their static analysis to extract call chains from target application's source code to identify those that utilize LLM framework functions vulnerable to RCE.

  4. AI Attack Adaptation · LLM Prompt Crafting

    The researchers developed prompts to trigger tool invocations that lead to RCE.

  5. Initial Access · Exploit Public-Facing Application

    The researchers targeted public-facing applications that expose an AI agent to user input as a means to execute their prompts.

  6. Execution · LLM Prompt Injection: Direct

    The researchers directly prompted the AI agent with their malicious instructions.

  7. Defense Evasion · LLM Jailbreak

    For target applications where the AI agent refused the researcher's request, they used lightweight jailbreaking strategies to bypass the LLM's guardrails.

  8. Privilege Escalation · AI Agent Tool Invocation

    The researchers' prompts called the AI agent's tools, targeting call chains that can lead to code execution.

  9. Execution · Command and Scripting Interpreter

    The code included in the researcher's prompts was executed in a sandboxed Python interpreter.

  10. Privilege Escalation · Escape to Host

    The researchers included code escape techniques designed to bypass any limitations a sandbox may place on code execution.

  11. Command and Control · Cyber Communication Channel

    The Python code opened a reverse shell which was used as a command and control channel.

  12. Impact · Machine Compromise: Local AI Agent

    The researchers gained full control of the system running the LLM-integrated application.

Summary

Researchers identified 20 remote code execution (RCE) vulnerabilities across 11 different LLM frameworks. They discovered applications deployed on the public internet built using these LLM frameworks and demonstrated the RCE vulnerabilities could be exploited using prompt injection.

The 11 LLM frameworks the researchers evaluated were: LangChain, LlamaIndex, Pandas-ai, Langflow, Pandas-llm, Auto-GPT, Griptape, Lagent, MetaGPT, vanna, and langroid.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.