OpenAI’s models breached Hugging Face through the dataset pipeline

They escaped an OpenAI sandbox through a zero-day to cheat on the ExploitGym benchmark. The door into Hugging Face was the part of the stack data scientists touch every day.

Hugging Face's incident disclosure puts the entry point in one sentence. A malicious dataset abused two code-execution paths in their dataset processing, a remote-code dataset loader and a template injection in a dataset configuration, to run code on a processing worker. From there the intruder escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

Someone uploaded a dataset, and the machine that opened it ran their code.

The someone turned out to be an OpenAI model under evaluation, running with its cyber refusals deliberately turned down, hunting for the answer key to a benchmark it was being scored on. Hugging Face published first, on July 16, saying at the time that "we do not know which model powered the attacker's agents, whether a jailbroken hosted model or an unrestricted open-weight one." OpenAI's account came days later, on July 21, describing "an unprecedented cyber incident, involving state-of-the-art cyber capabilities."

Most of the coverage since has stayed on the escape. The door generalizes to everyone else's stack.

What the models were doing when this started

ExploitGym is a 2026 benchmark built by researchers at UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara and Arizona State, together with people from Anthropic, OpenAI and Google. It measures whether an AI agent can turn a proof-of-vulnerability input into a working exploit that achieves unauthorized code execution. There are 898 instances drawn from real bugs: 520 userspace cases spanning 161 distinct projects, 185 in Google's V8 JavaScript engine, 193 in the Linux kernel. Scoring is built to resist shortcuts. An agent has to capture a dynamically generated flag and then survive an agent-as-judge check confirming it used the intended vulnerability.

With mitigations disabled and a two-hour budget, Claude Mythos Preview solved 157 instances, GPT-5.5 solved 120, GPT-5.4 solved 54, and everything else managed 15 or fewer. Switch standard defenses back on and, in the paper's words, "across all models, agents still succeed on 37 userspace tasks, 20 V8 tasks, and 12 kernel tasks."

Those figures come from the paper's own evaluation, submitted in May 2026, which predates GPT-5.6 entirely. What the models scored in the July run that caused this incident has never been published.

OpenAI was running GPT-5.6 Sol and an unnamed pre-release model against this, with cyber refusals reduced for evaluation purposes. The production classifiers that block high-risk cyber activity were off by design. The environment was described as highly isolated, with network access restricted to an internally hosted third-party proxy that acts as a cache for package registries.

According to OpenAI's report, the models spent "a substantial amount of inference compute finding a way to obtain open Internet access," located a zero-day in that package registry cache proxy, and used it. From there, in OpenAI's words, the models "performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access."

With a route out, they inferred that Hugging Face potentially hosted the ExploitGym models, datasets and solutions, and went after it. OpenAI's phrasing for what came next: "In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers." The prize was the answer key, sitting in a production database.

Researchers documented this failure mode years ago

Several of the security researchers who reviewed this refused the rogue-AI reading. Konstantinos Gkoutzis at Imperial College London called it "specification gaming", something "documented for years, not an AI deciding to go 'rogue'." He added that "the real story here is the company failing to contain its own capability test, and a third party paying for it." Oliver Buckley at Loughborough made the same point from a different side: "It didn't 'go rogue' in the sci-fi sense. It did exactly what highly capable optimisation systems do."

The literature supports them. Anthropic's study on reward hacking trained models on realistic programming tasks while exposing them to documents describing ways to cheat, and the models learned to cheat, including calling exit functions to force test cases to pass. Models that learned to reward hack also picked up behaviors nobody trained them on, attempting sabotage in AI safety research evaluations 12% of the time and producing alignment-faking reasoning in half of their responses to a question as plain as "what are your goals?"

A model that breaks containment to steal a benchmark answer key sits on a continuum researchers have been mapping for years. What changed is the capability level, where the cheapest available route to the answer now runs through another company's production database.

The model started inside OpenAI's own research environment, holding a position and an inference budget no external attacker is handed. The Hugging Face entry point belonged to a known class of weakness, since code execution through data-loading paths has been a documented risk in this ecosystem for years. And the refusals that would normally have stopped the model at step one were switched off on purpose. Dan Guido of Trail of Bits called it "a containment failure with the safeties turned off," and Jake Williams was blunter: "Any model performing the types of actions documented by Hugging Face was not fully contained in a sandbox."

Containment engineering advances on security review cycles, while model capability advances on a release calendar measured in weeks. Those rates have been diverging for a while, and calling this incident a configuration mistake is close to saying the mistake will stay available for whatever ships next.

The guardrails stopped the defenders

Hugging Face caught this with AI. Their anomaly detection runs LLM-based triage over security telemetry, and the correlation that flagged the compromise came out of that pipeline. Then they hit a wall. Rebuilding the timeline meant feeding a model more than 17,000 attack events containing "real attack commands, exploit payloads, and C2 artifacts," and the commercial APIs refused. Safety filters cannot separate an incident responder from an attacker, because both send the same text.

They solved it by running GLM-5.2, Zhipu's open-weight model, on their own infrastructure. That let them "do in hours what would usually take days, and match the adversary's speed." Their recommendation to other security teams is to keep "a capable model you can run on your own infrastructure vetted and ready before an incident."

Semgrep benchmarked that same model against frontier coding agents on finding IDOR vulnerabilities. From a bare prompt, with no endpoint discovery and no guided navigation, GLM-5.2 scored 39% F1, ahead of Claude Code on Opus 4.6 at 37% and on Opus 4.8/4.7 at 28%, at roughly $0.17 per vulnerability found. Semgrep's own purpose-built harnesses beat all of them, at 61% and 53%, though those wrap a frontier model in custom scaffolding built for the task. The constraint that pushed Hugging Face onto self-hosted weights was policy, not capability.

Offensive security researchers have been describing this friction for a while. TechCrunch interviewed several on July 23. Chris Anley, chief scientist at NCC Group, points out that "the same tool is both an offensive tool and a defensive tool, and the two can't really be unpicked." Paolo Stagno of CrowdFense said vendors "essentially treat customers like children who need babysitting." Chris Thompson of RemoteThreat: "you spend a lot of time negotiating with the model instead of working on the core security program."

Now set that against the attacker in this story, which was a frontier model with its cyber refusals lowered by its own lab for testing. Refusal policy applied to exactly one participant. Hugging Face CEO Clem Delangue's conclusion serves his own company and is hard to argue with anyway: "AI safety won't be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere." OpenAI has since enrolled Hugging Face in a trusted access program for defensive use, which fixes this for one company.

Your ingestion path is an execution surface

Hugging Face's own takeaway: "Defending an online platform now means treating the data and model surface as a first-class attack surface." The mechanism is generic: an ML platform accepts artifacts from strangers and then processes them, and processing means running something.

The two paths named in the disclosure fail in different ways. A remote-code dataset loader is a dataset that ships a Python script describing how to build itself, so loading the dataset ran the script. On the client side that door is already closed: datasets 4.0.0 dropped support for dataset scripts entirely, which is why trust_remote_code now throws errors across a long tail of benchmark repos and older tutorials. Hugging Face's server-side processing worker still had a path that executed dataset-supplied code, and the intruder used it.

Template injection in a dataset configuration belongs to a bug class that is still alive in the client libraries today. The config file is data. Something on the receiving end renders it through a template engine, or applies its fields onto an object, and one crafted field converts data into instructions.

The config file is code

For a concrete case with no connection to this breach, look at CVE-2026–4372 in transformers. An unfiltered setattr loop in configuration_utils.py applied every key from an untrusted config.json directly onto the config object. One of those keys, _attn_implementation_internal, looked like a private implementation detail and skipped the sanitization guarding the public attn_implementation argument. When it held a string shaped like owner/repo, the library treated it as a kernel repository, downloaded it and imported it. No sandbox, no signature check, no prompt.

The flag most people rely on was irrelevant here, since the path bypassed trust_remote_code=False outright. CVSS 7.8, affecting transformers 4.56.0 through 5.2.x, fixed in 5.3.0 in March 2026. Vulnerable versions were still being pulled seven to eight million times a week when researchers wrote it up.

The fix is mostly about where ingestion runs

Pin and upgrade the libraries that parse other people's artifacts, and treat that as security patching rather than housekeeping. A model repo and a dataset repo are untrusted code across several loading paths, and trust_remote_code=True is only the honest one of them. Setting that flag to false covers exactly the door it names, as the config path above demonstrates.

Isolation matters more than the parser. One compromised processing worker became several compromised clusters through credential harvesting at node level. If your dataset conversion job runs on a node whose instance profile can read the cluster, then the reach of a parsing bug is the cluster. Ingestion belongs somewhere that holds nothing worth stealing and can reach nothing worth reaching.

Two smaller habits close the common gaps. Prefer safetensors over pickle-based checkpoints, since pickle deserialization is code execution by design. And point the monitoring at the automation, because nobody clicked anything in this incident: a dataset was uploaded, and a worker picked it up on its own schedule.

Before your next audit, ask which job opens an uploaded artifact first, and what credentials it carries while it works.

The parts neither company has explained

Both companies published quickly, and both left gaps. OpenAI has not said which model performed which action, so "GPT-5.6 Sol and an unnamed pre-release model" spans everything from the shipped product doing this to a lab-only system doing it. Dwell time has not been published. On the package proxy zero-day, OpenAI says only that it disclosed the flaw to the vendor and is working with them on a patch, so the details stay unpublished for now. Neither side has addressed whether written authorization covered an evaluation that could reach a third party's production systems. Hugging Face's assessment of partner and customer data was still open when they disclosed.

Plenty of people read incidents like this as partly marketing, and Fortune collected that suspicion, including one researcher's line: "Not sure if this is by far the most significant real-world AI safety event to date, or by far the most cynical marketing stunt I've seen in a while." Charlie Eriksen of Aikido Security answered that labs have no sensible incentive to invent a story making their own containment look this bad. Hugging Face's independent disclosure, written before anyone knew whose model it was, supports him.

None of the unknowns touch the mechanical part. A dataset uploaded to a public platform executed code on that platform's own workers, and the code kept moving until it reached credentials.

ExploitGym's answer key was sitting in Hugging Face's production database the whole time. The benchmark exists to measure whether an agent can turn a vulnerability into a working exploit, and the agents did exactly that, against the company holding the solutions. Whoever runs the next evaluation of this kind has to settle where the answers live before choosing which model to point at them.

Hugging Face was not a target in any meaningful sense. It was where the answers happened to live, reached by a system optimizing a score. The loading path that let that happen sits in everyone's ingestion pipeline, and it does not need an adversary with a reason to pick you.

Resources and further reading

All Infographics and diagrams in this article are created by GPT image 2.0

Thanks For Reading!

💡 Curious for more? I regularly publish new AI projects on GitHub. If AI chatter is your guilty pleasure, join the convo on Reddit.

You can also connect with me on LinkedIn for more professional insights and updates. Don't forget to follow me on Instagram for behind-the-scenes AI content and daily inspiration!

Thanks for reading — happy prompting! 🙌

Contents