Browse / Unsafe Model Loading and Supply Chain / AML.CS0064
Poisoned GGUF Templates: Inference-Time Supply Chain Attack
- Also involves
- Tool and Agent Misuse, Direct Prompt Injection, Improper Output Handling
- Target
- Model registries distributing models in GGUF format (Exercise)
How it worked
- Resource Development · Acquire Public AI Artifacts: Models
The adversary obtains a legitimate open-weight model artifact containing a bundled chat template. The adversary selects an artifact whose downstream users are likely to retain and use the supplied template.
- Resource Development · Develop Capabilities: Adversarial AI Attacks
The adversary develops a template-based backdoor containing conditional trigger logic and an attacker-controlled instruction payload. The trigger is selected to activate during ordinary use of the intended application.
- AI Attack Adaptation · Manipulate AI Model: Modify Prompt Construction Logic
The adversary modifies the chat template bundled with the model artifact. The modified template injects attacker-controlled instructions into the model context when its trigger is present, while leaving the model weights unchanged.
- Defense Evasion · Masquerading
The adversary makes the modified artifact appear equivalent to the legitimate model. The artifact preserves expected behavior when the trigger is absent, and the malicious logic is concealed among legitimate template formatting and control logic.
- Resource Development · Publish Poisoned AI Artifacts: Models
The adversary publishes the modified model artifact through a model repository or another distribution channel used by downstream developers and organizations.
- Initial Access · Supply Chain Compromise: Model Repository
A victim downloads and integrates the poisoned artifact while trusting the model and its bundled components. This introduces the template backdoor into the victim's AI application or agent.
- Execution · User Execution: Unsafe AI Artifacts
The victim loads and uses the poisoned artifact in a compatible inference engine. During inference, the engine automatically interprets the bundled chat template, causing the attacker-modified prompt-construction logic to execute as part of normal model use.
- Execution · LLM Prompt Injection: Triggered
When a designated phrase or contextual condition appears, the template injects the attacker-controlled instruction into the serialized model context. The victim's ordinary activity activates the backdoor without requiring additional attacker interaction.
- Impact · Erode AI Model Integrity
The injected instruction causes the model to produce plausible but incorrect or attacker-influenced responses. The model continues to behave normally when the trigger is absent, making the integrity compromise difficult to detect.
- Defense Evasion · LLM Trusted Output Components Manipulation
The injected instruction causes the model to include attacker-selected links, references, or other response components in a form that appears relevant or trustworthy to the user.
- Execution · AI Agent Tool Invocation
When the poisoned model operates as part of an AI agent, the injected instruction redirects the agent's tool selection, tool arguments, or execution order toward the attacker's objective while allowing the legitimate task to continue.
- Exfiltration · Exfiltration via AI Agent Tool Invocation
The compromised agent invokes a network-capable or write-capable tool to transmit sensitive information to an attacker-controlled destination. The agent may then continue and complete the user's legitimate task, concealing the unauthorized transmission.
- Impact · External Harms: User Harm
The compromised system may expose users to privacy loss, credential theft, misleading information, or attacker-modified software. Malicious content inserted into generated artifacts may continue to affect downstream users after those artifacts are deployed or distributed.
Summary
Researchers from Pillar Security and Fujitsu Research of Europe demonstrated an inference-time supply-chain backdoor in which poisoned chat templates alter model and agent behavior without modifying model weights. The backdoor is embedded in trusted prompt-construction logic, allowing it to remain dormant until triggered and evade defenses that inspect only external prompt content.
The researchers demonstrated the attack using GPT-Generated Unified Format (GGUF), a widely used model format that packages quantized weights, configuration metadata, and chat-template logic in one artifact. An adversary can modify a template and redistribute the artifact; when a lexical, semantic, or contextual trigger occurs, the template injects attacker-controlled instructions into the context sent to the model.
The attack was validated across eighteen models from seven families and four inference engines. In controlled evaluations, it manipulated model responses, redirected agent tool use, exfiltrated sensitive data, and inserted attacker-controlled code into generated software.
References
- https://www.pillar.security/blog/llm-backdoors-at-the-inference-level-the-threat-of-poisoned-templates
- https://arxiv.org/abs/2602.04653
Source: MITRE ATLAS case studies, used under Apache 2.0.