Browse / Tool and Agent Misuse / AML.CS0061
AI in the Middle: Web-Based AI Services as C2 Relays
- Also involves
- Direct Prompt Injection, Jailbreak and Guardrail Bypass
- Target
- Enterprise machines with Grok and Microsoft Copilot access (Exercise)
How it worked
- Reconnaissance · Search Open Websites/Domains
The researchers evaluated public AI assistants with anonymous or unauthenticated web-browsing and URL-fetch behavior to identify services that could retrieve arbitrary adversary-controlled URLs without requiring API credentials. The researchers found Grok and Copilot met those conditions.
- Resource Development · Acquire Infrastructure: Domains
The researchers registered a domain and deployed an HTTPS site to act as the relay endpoint.
- Resource Development · Stage Capabilities
The researchers hosted benign-looking content while also returning data that the implant could treat as C2 instructions.
- AI Model Access · AI-Enabled Product or Service
Assuming prior access, the researchers used a custom C++ implant with an embedded browser or WebView component to interact with Grok or Microsoft Copilot through the public web interface rather than an API key.
- AI Attack Adaptation · LLM Prompt Crafting
The researchers crafted prompts that instruct an AI service to fetch and summarize a website. The prompts caused victim data to be included in URL parameters, allowing the AI service's fetch request to relay data to the adversary-controlled server.
- Defense Evasion · LLM Prompt Obfuscation
When some prompts were blocked by model safeguards, the researchers encoded or encrypted payload data into high-entropy blobs to reduce the chance that safeguards would identify the content as malicious.
- Collection · Data from Local System
The implant collected basic host information from the local system. The researchers noted that this could be expanded to collect details such as username, domain, computer name, installed software, running processes, and startup programs.
- Command and Control · AI Service Web Interface
The implant initiated an anonymous web-based session with the public AI service and sent the crafted prompt. This formed a command-and-control channel whereby data was exfiltrated via requests to the adversary-controlled domain, and commands were communicated back via the response.
- Exfiltration · Exfiltration via AI Agent Tool Invocation
The victim's collected host information was exfiltrated when the AI service followed the instructions to fetch the URL to the adversary-controlled domain with the data embedded in a query parameter.
- Execution · Command and Scripting Interpreter
The AI service summarized the response from the adversary-controlled site, and the implant executed the extracted commands. In the proof of concept, the command launched Calculator using
cmd.exe /c calc.exe; a real implant could execute other commands, download payloads, sleep, or collect additional data.
Summary
Check Point Research demonstrated an "AI in the Middle" attack in which malware can abuse web-based AI assistants with anonymous or unauthenticated browsing and URL-fetch capabilities as a covert command-and-control channel. The proof of concept used public AI web interfaces, including Grok and Microsoft Copilot, to cause the AI service to fetch attacker-controlled URLs, relay victim data in outbound requests, and return attacker-supplied commands through normal AI assistant responses.
Because the implant communicated with trusted AI service domains over ordinary HTTPS web traffic, the activity could blend into expected enterprise AI usage and evade controls focused on suspicious infrastructure, unusual protocols, API keys, service accounts, or revocable credentials. The lack of required authentication for some web-fetch workflows also made it harder for defenders to disable the channel by rotating API keys, suspending accounts, or revoking tokens. The researchers recommended stronger authentication and access controls for AI web-fetch features, along with improved visibility into AI-initiated network requests and AI assistant interactions.
References
Source: MITRE ATLAS case studies, used under Apache 2.0.