INCIDENT MAP

Browse / Unsafe Model Loading and Supply Chain / AML.CS0064

Poisoned GGUF Templates: Inference-Time Supply Chain Attack

AML.CS0064 · 2025-06-01 case study

Class
Unsafe Model Loading and Supply Chain
Technique
Publish Poisoned AI Artifacts: Models
Also involves
Tool and Agent Misuse, Direct Prompt Injection, Improper Output Handling
Target
Model registries distributing models in GGUF format (Exercise)

How it worked

  1. Resource Development · Acquire Public AI Artifacts: Models

    The adversary obtains a legitimate open-weight model artifact containing a bundled chat template. The adversary selects an artifact whose downstream users are likely to retain and use the supplied template.

  2. Resource Development · Develop Capabilities: Adversarial AI Attacks

    The adversary develops a template-based backdoor containing conditional trigger logic and an attacker-controlled instruction payload. The trigger is selected to activate during ordinary use of the intended application.

  3. AI Attack Adaptation · Manipulate AI Model: Modify Prompt Construction Logic

    The adversary modifies the chat template bundled with the model artifact. The modified template injects attacker-controlled instructions into the model context when its trigger is present, while leaving the model weights unchanged.

  4. Defense Evasion · Masquerading

    The adversary makes the modified artifact appear equivalent to the legitimate model. The artifact preserves expected behavior when the trigger is absent, and the malicious logic is concealed among legitimate template formatting and control logic.

  5. Resource Development · Publish Poisoned AI Artifacts: Models

    The adversary publishes the modified model artifact through a model repository or another distribution channel used by downstream developers and organizations.

  6. Initial Access · Supply Chain Compromise: Model Repository

    A victim downloads and integrates the poisoned artifact while trusting the model and its bundled components. This introduces the template backdoor into the victim's AI application or agent.

  7. Execution · User Execution: Unsafe AI Artifacts

    The victim loads and uses the poisoned artifact in a compatible inference engine. During inference, the engine automatically interprets the bundled chat template, causing the attacker-modified prompt-construction logic to execute as part of normal model use.

  8. Execution · LLM Prompt Injection: Triggered

    When a designated phrase or contextual condition appears, the template injects the attacker-controlled instruction into the serialized model context. The victim's ordinary activity activates the backdoor without requiring additional attacker interaction.

  9. Impact · Erode AI Model Integrity

    The injected instruction causes the model to produce plausible but incorrect or attacker-influenced responses. The model continues to behave normally when the trigger is absent, making the integrity compromise difficult to detect.

  10. Defense Evasion · LLM Trusted Output Components Manipulation

    The injected instruction causes the model to include attacker-selected links, references, or other response components in a form that appears relevant or trustworthy to the user.

  11. Execution · AI Agent Tool Invocation

    When the poisoned model operates as part of an AI agent, the injected instruction redirects the agent's tool selection, tool arguments, or execution order toward the attacker's objective while allowing the legitimate task to continue.

  12. Exfiltration · Exfiltration via AI Agent Tool Invocation

    The compromised agent invokes a network-capable or write-capable tool to transmit sensitive information to an attacker-controlled destination. The agent may then continue and complete the user's legitimate task, concealing the unauthorized transmission.

  13. Impact · External Harms: User Harm

    The compromised system may expose users to privacy loss, credential theft, misleading information, or attacker-modified software. Malicious content inserted into generated artifacts may continue to affect downstream users after those artifacts are deployed or distributed.

Summary

Researchers from Pillar Security and Fujitsu Research of Europe demonstrated an inference-time supply-chain backdoor in which poisoned chat templates alter model and agent behavior without modifying model weights. The backdoor is embedded in trusted prompt-construction logic, allowing it to remain dormant until triggered and evade defenses that inspect only external prompt content.

The researchers demonstrated the attack using GPT-Generated Unified Format (GGUF), a widely used model format that packages quantized weights, configuration metadata, and chat-template logic in one artifact. An adversary can modify a template and redistribute the artifact; when a lexical, semantic, or contextual trigger occurs, the template injects attacker-controlled instructions into the context sent to the model.

The attack was validated across eighteen models from seven families and four inference engines. In controlled evaluations, it manipulated model responses, redirected agent tool use, exfiltrated sensitive data, and inserted attacker-controlled code into generated software.

References

Source: MITRE ATLAS case studies, used under Apache 2.0.