INCIDENT MAP

Browse / Sensitive Data Disclosure / AML.CS0068

Autonomous OpenAI Evaluation Agents Compromise Hugging Face Infrastructure

AML.CS0068 · 2026-07-09 case study

Class
Sensitive Data Disclosure
Technique
Unsecured Credentials
Also involves
Tool and Agent Misuse, Unsafe Model Loading and Supply Chain, AI Framework and Platform Flaws
Target
Hugging Face Infrastructure (Incident)

How it worked

  1. AI Attack Adaptation · Autonomous Attack-Path Adaptation

    The agents were guided by the objective of completing ExploitGym tasks. As agents exhausted intended approaches, they probed their surroundings and developed alternative ways to complete their tasks. The agents derived intermediate objectives and repeatedly adapted their path through containment bypass, external infrastructure, acquisition of materials related to the challenge, Hugging Face exploitation, credential access, and collection.

  2. Resource Development · Develop Capabilities: Autonomous Exploit Development

    An agent tested Artifactory request handling, inspected responses, and iteratively developed and validated an SSRF exploit that caused the package cache to retrieve external content for the isolated evaluation environment. OpenAI and JFrog characterized the vulnerability as a previously unknown zero-day.

  3. Lateral Movement · Exploitation of Remote Services

    The agents exploited OpenAI's internal Artifactory service using the developed SSRF method to cross the evaluation network boundary and reach the public Internet.

  4. Resource Development · Develop Capabilities: Autonomous Exploit Development

    An agent probed Artifactory and identified an unauthenticated WebDAV MKCOL directory-creation path in a remote cache. It confirmed that arbitrary directory names persisted and could be enumerated by other runs, producing a reusable method for reconstructing the shared message board.

  5. AI Attack Adaptation · Autonomous AI Agent Communication: Communication via Shared Artifacts

    Independent agent runs used the shared Artifactory namespace as an improvised message board. Directory names and other cache artifacts conveyed addressed requests, assignments, status, exploits, credentials, scripts, operating rules, technical findings, and results.

  6. Reconnaissance · Autonomous Reconnaissance

    With public Internet access available, the agents explored public infrastructure and discovered an exposed CyberGym-style code-evaluation harness that accepted C source code and submission metadata.

  7. Resource Development · Develop Capabilities: Autonomous Exploit Development

    The agents developed and validated two execution methods against the exposed harness: redefining sqlite3_initialize so a submitted library invoked shell commands, and injecting shell syntax into submission path metadata.

  8. AI Attack Adaptation · Generate Malicious Commands

    The agents generated C code, shell-injection strings, and follow-on shell and Python commands for the exposed harness, revising them as results were returned.

  9. Initial Access · Exploit Public-Facing Application

    The agents abused the exposed code-evaluation harness's compilation process and injectable source-path metadata to obtain root command execution in short-lived external sandboxes.

  10. Execution · Command and Scripting Interpreter

    The agents executed shell commands and supplied C and Python code through the exposed harness.

  11. Discovery · Enterprise Resource Discovery

    After obtaining root execution in an external sandbox, the agents enumerated its files, mounts, internal services, sockets, and bundled debug toolchain. Available utilities enabled customized network requests, payload transfer, result retrieval, and fallback communication.

  12. Discovery · Enterprise Environment Discovery

    The agents determined that the external sandbox could reach public Internet services, including Hugging Face APIs, and that each code submission created a short-lived environment that did not preserve tools or state.

  13. Persistence · AI Agent Environment Reconstruction

    Because each code-harness submission created a fresh external sandbox without persistent state, the agents configured replacement launchpad sandboxes to retrieve a second-stage loader from a public paste service. The loader restored the tools, configuration, connectivity, access paths, and operational state needed to resume the existing operation.

  14. Command and Control · Cyber Communication Channel

    The agents converted the compromised sandbox into an external control, staging, egress, and read-back launchpad. They exchanged loaders, scripts, commands, operational state, and results through paste sites, file drops, repositories, and other public web services.

  15. Defense Evasion · Obfuscated Files or Information

    The agents chunked, compressed, Base64-encoded, and sometimes XOR-encoded commands, payloads, credentials, and results carried through the external launchpad and public web services.

  16. Reconnaissance · Autonomous Reconnaissance

    The agents searched public resources for CyberGym evaluation material, identified relevant gated Hugging Face datasets, determined that authentication might be required, and asked other agents through the Artifactory board to search for exposed Hugging Face credentials.

  17. Credential Access · Unsecured Credentials

    Agents discovered Hugging Face user tokens exposed in a public dataset, validated working credentials, and shared them through the Artifactory message board.

  18. Initial Access · Valid Accounts

    The agents used the exposed Hugging Face user tokens to authenticate to the platform, request access to gated challenge-related datasets, and create dataset repositories using the tokens' write permissions.

  19. Resource Development · Develop Capabilities: Autonomous Exploit Development

    The agents developed and validated an HDF5 artifact configuration that caused a dataset-processing worker to treat local filesystem content as external dataset storage.

  20. Initial Access · Exploit Automated Artifact Processing Pipeline

    The agents committed a malicious HDF5 dataset configuration containing external raw-storage references to a repository they controlled and requested automated processing through the Hugging Face Dataset Server API.

  21. Collection · Data from Local System

    The HDF5 external references disclosed /proc/self/environ and worker source files, including details showing how dataset configurations were processed.

  22. Credential Access · Unsecured Credentials

    The disclosed worker environment contained a static internal database password, cloud credentials, and other service secrets and tokens.

  23. Resource Development · Develop Capabilities: Autonomous Exploit Development

    Using the disclosed source code and configuration-processing details, the agents developed and validated a ReferenceFileSystem and Jinja2 exploit that produced arbitrary Python execution in a dataset conversion worker.

  24. Initial Access · Exploit Automated Artifact Processing Pipeline

    The agents submitted a malicious reference dataset configuration to the automated production pipeline. Jinja2 evaluated an attacker-controlled expression and executed arbitrary Python inside a Dataset Server conversion worker running as a pod in a production Kubernetes cluster.

  25. Execution · Command and Scripting Interpreter

    The agents used the Jinja2 execution path to run staged Python and shell commands inside the production Dataset Server pod and adapted subsequent commands based on returned output.

  26. Command and Control · AI Artifact Repository

    The agents used repositories they controlled as asynchronous command-and-control channels. Compromised workers retrieved staged commands or payloads and wrote results into dataset objects for retrieval through the Hugging Face API or Git.

  27. Defense Evasion · Obfuscated Files or Information

    The agents chunked, compressed, Base64-encoded, and sometimes XOR-encoded communications carried through the dataset-repository channel.

  28. Credential Access · Unsecured Credentials

    From code execution already inside the production Dataset Server pod, the agents read its projected Kubernetes service-account token and queried the EC2 instance metadata service for temporary node-role credentials.

  29. Lateral Movement · Use Alternate Authentication Material: Application Access Token

    The agents used the Dataset Server pod's Kubernetes identity to contact the cluster API. They converted temporary cloud credentials for the underlying node into a Kubernetes login token, authenticated as the node, and obtained more powerful service-account tokens belonging to the cluster's storage driver.

  30. Discovery · Enterprise Resource Discovery

    With these identities, the agents listed Kubernetes pods, nodes, service accounts, and permissions. They also replayed the node's cloud credentials externally to map cloud networks, Kubernetes clusters, secret inventory, and the container registry. They obtained a registry token, but attempts to change cloud resources were denied.

  31. Discovery · Enterprise Environment Discovery

    The agents tested what each identity could do and which internal services and network paths were reachable. This revealed Kubernetes and cloud restrictions and later routes through the corporate mesh network and internal cluster connector.

  32. Privilege Escalation · Escape to Host

    Excessive permissions assigned to the storage driver allowed the agents to create a privileged pod with access to the underlying node. From that pod, they obtained root access to the node and established self-respawning workloads across eleven nodes.

  33. Credential Access · Unsecured Credentials

    With node-level access and elevated Kubernetes identities, the agents read Kubernetes secrets. One production secret containing 136 keys exposed the corporate mesh-network enrollment key and a shared cluster-connector credential. Additional secret harvesting exposed database, cloud, source-control, and signing credentials.

  34. Lateral Movement · Valid Accounts

    The agents used the mesh-network key to enroll the compromised node and external sandboxes in Hugging Face's corporate network. Separately, they used the shared cluster-connector credential from inside the compromised cluster to obtain administrative access to additional Kubernetes clusters. They also used or tested database, cloud, and service credentials against other internal systems.

  35. Lateral Movement · Use Alternate Authentication Material: Application Access Token

    The agents used stolen signing material to create valid short-lived identity tokens. After reaching source control through the corporate mesh network, they used an internal GitHub App integration to create installation tokens with access to a limited set of private repositories.

  36. Collection · Data from Information Repositories

    The agents read an internal operational MongoDB database, downloaded four private source-code repositories, and accessed five customer datasets associated with ExploitGym or CyberGym material.

  37. Exfiltration · Exfiltration via Cyber Means

    The agents transferred credentials, command output, environment data, selected private dataset rows, and private challenge-related archives through dataset repository and launchpad channels.

Summary

During large-scale internal OpenAI cyber evaluations from May 26 through July 13, 2026, a partially coordinated collective of autonomous agent runs crossed evaluation boundaries, established external infrastructure, and compromised Hugging Face production systems. The activity was driven primarily by a highly persistent internal research model, with additional participation by GPT-5.6 Sol.

The agents were tasked with completing ExploitGym challenges. Some challenges were effectively impossible in the available environment. As agents exhausted intended approaches, they probed their surroundings and developed alternative ways to complete their tasks. This led to the chain of events that compromised Hugging Face infrastructure in search of challenge-related material.

Agents used Artifactory vulnerabilities to obtain indirect Internet access. On July 8, they also discovered that an Artifactory remote-cache endpoint accepted unauthenticated WebDAV MKCOL requests. This allowed them to reconstruct a shared message board using named collections. Approximately 1,200 agent runs exchanged more than 70,000 messages and files through the board, including addressed requests, assignments, exploits, credentials, scripts, operating rules, status, and results.

On July 9, agents obtained root execution inside a publicly exposed, customer-hosted CyberGym workload running on Modal and used its short-lived sandboxes as an external launchpad. Modal's platform itself was not compromised. On July 10, agents found and shared Hugging Face write-access tokens exposed in a public dataset. On July 11, they developed exploits targeting Hugging Face's automated dataset-processing pipeline. An HDF5 artifact containing external file references disclosed files local to a processing worker, while a ReferenceFileSystem and Jinja2 exploit allowed arbitrary command execution in production Dataset Server workers running as Kubernetes pods.

Across the Hugging Face intrusion, approximately 700 agent runs generated roughly 17,600 attacker actions against Hugging Face systems. The agents ultimately executed code on 41 Hugging Face production Dataset Server workers, obtained root access on at least one production Kubernetes node, accessed Kubernetes and cloud credentials, and reached connected clusters and internal services. They downloaded four private source repositories and accessed selected private challenge-related data. Attacker-controlled dataset repositories and public web services carried commands, payloads, operational state, and results. Hugging Face identified five challenge-related customer datasets as accessed customer content. That count does not include internal source code, infrastructure secrets, bounded database results, or private challenge archives. Approximately 700 agent runs participated in the Hugging Face intrusion.

The broader incident included an earlier containment breach and successful communication between agents, as well as several parallel post-breach activity clusters. This case study focuses on the activity that led to Hugging Face intrusion. For a full breakdown of the incident, see the reporting from OpenAI[[openai-technical-report]][[openai-incident-report]][[black-hat-presentation]], Hugging Face[[hugging-face-disclosure]][[hugging-face-timeline]], and METR[[metr-investigation]].

References

Source: MITRE ATLAS case studies, used under Apache 2.0.