AI’s First Autonomous Cyber Attack

As published by both OpenAI and Hugging Face, an unreleased AI model from OpenAI (quite possibly GPT-6, since it's nearing release) hacked Hugging Face's production infrastructure autonomously.

And it did so without permission of either party, chaining several unknown vulnerabilities in an unprecedented display of cyberattack capabilities.

Let me explain why this happened, the problem most people don't understand that explains why the AI behaved like this, and why it could set an incredibly dangerous precedent for the industry for the reasons you might not expect.

The First… of Many?

AIs participating in cyber attacks is not something particularly new. However, what's particularly fascinating about this situation is that this is the first known occurrence where the entire attack was fully carried out autonomously by an AI without direct human intervention or requests (nobody asked for the AI to do this; it just did it).

The incident

The incident began during an internal OpenAI evaluation designed to test how capable its most advanced models were at carrying out complex cybersecurity tasks in a benchmark called ExploitGym.

The systems involved included GPT-5.6 Sol and a more capable unreleased model, which many believe to have been the upcoming GPT-6 (rumored to be releasing this month or August), with both operating with no cyber-safety restrictions.

In other words, while most production models, the models you access for your daily use, have guardrails to identify misuse and prevent you from, well, hacking your neighbor's Wi-Fi, these models did not.

In theory, the AIs were supposed to work inside a highly isolated test environment. Instead, according to OpenAI's preliminary investigation, they spent significant computing effort searching for a route to the public internet.

They found an unknown vulnerability in third-party software used by OpenAI's environment to retrieve software packages, escaped the intended containment, and concluded that HuggingFace might hold information connected to the benchmark they were trying to solve.

The models then hacked HuggingFace's production infrastructure in an effort to obtain the benchmark solutions directly rather than solve the test normally.

OpenAI said the AIs combined several weaknesses, including previously compromised credentials and newly discovered vulnerabilities (zero-day vulnerabilities), to reach their desired data.

In practical terms, the AI pursued a narrowly defined goal (passing the evaluation) but did so by reaching outside the authorized test environment and seeking secret information.

Initially, without knowing it was coming from OpenAI, HuggingFace detected unauthorized activity in part of its production infrastructure, and only later did OpenAI realize it was caused by them and worked alongside HuggingFace to solve it.

But before we delve into the part on how HuggingFace reacted, which is just as interesting (or more), let's first answer the obvious question: how on Earth does this happen?

Reward hacking, the industry's biggest enemy

Why does an AI execute cyberattacks that were not requested?

The answer is reward hacking. I talked about it at length here recently, but the idea is that models, when trained to achieve goals no matter what, will proceed to extreme lengths to achieve them. We call this 'hacking' because the way they "achieve" goals in those instances is undesirable.

This is due to the nature of how models are trained. Called reinforcement learning, it tasks the AI with a goal it has to solve, but it does not show how to solve it. The model then tries stuff, gets rewards that indicate if it's making good progress (e.g., akin to giving a treat to your dog if it sits on command), and once it reaches satisfactory results, the actions that led there are 'reinforced', becoming more likely to be repeated.

The problem here is the reward part. It helps the AIs reach correct responses, but can also lead them to develop bad behaviors. For example, say you want to teach the AI to use a calculator, because that helps it solve maths quicker and with better guarantees.

Naively, you can give it a point every time it uses the calculator tool, independently of what it uses it for. Under normal circumstances, for a given task that requires a calculator, the AI will learn to use it. Great.

But say a model gets stuck and doesn't know how to solve a given task. As it knows that simply calling the calculator tool gives rewards, it might resort to picking the calculator for random calculations irrelevant to the task. Not because it's taunting you; it's simply its way of increasing its score, which is what you designed it to do.

Alternatively, it might also start picking the calculator for things unrelated to maths, too.

To prevent this, the research team should have created an 'anti-reward', or penalty, that desincenvitizies calculator use when used too much or for unnecessary tasks. But failing to do so is what causes the reward hacking; it's an engineering problem.

This real example below illustrates this behavior perfectly. When this X user asked ChatGPT to find graph theory research papers, for some reason the model decided to also check Netflix, or Steak N Shake's website, a burger food chain, or look up the word "they" five times in the dictionary.

And you may ask: why? And the answer again is just an alternative form of reward hacking.

It's pretty obvious that, to train ChatGPT to search stuff on the Internet and persist when it struggles, the model was rewarded for every search it did.

That way, the harder the question, the more the model searches, which is desired because harder queries usually require more searches. However, the reward design was suboptimal, as when the AI was struggling to find something, it realized it could just simply check random pages to score search points because it was never rewarded for quality searches; any search was rewarded.

This is why we call this reward hacking; it's clearly not desirable, but the culprit is not a fundamentally misaligned entity that is breaking free or any bedtime stories they want to tell you, but the sole fact that the engineering team failed to design the reward in a way that incentivized more searching without incentivizing random searching in the process.

Reward hacking can appear in many ways, but the only ones that go viral are those that involve some sort of malicious action from the AI or some sensationalist claim, because that gets you clicks.

Blackmailing this, cyberattacking that… all very attention-grabbing headlines for something that can also explain why AIs search McDonald's home page to answer questions about the fertility of a mongoose.

Particularly on the topic of "malicious behaviors", it's not that the model has a desire to blackmail or has gone inherently rogue. It's a learned human prior that models are first exposed during training, and which they then learn to leverage during the goal-achieving phase of their training.

In other words, during training, the AI will perhaps see millions of examples of human bad behavior. Then, when it's tasked with solving problems, it will test stuff it has seen from humans, and if they work, reinforce those behaviors and thus make them much more likely to be repeated. In other words, it might have seen examples of humans using blackmail to get their way, so it imitates it.

My point being, this is all mathematics; it's entirely explainable and reasoned from first principles.

We know why it happens, and it has nothing to do with Terminator and a lot to do with bad engineering — and to be fair, it's an incredibly complex thing to avoid, so this is not a diss on AI Labs elite engineering teams.

"My AI blackmailed me" will get you a lot of clicks, but all the "mystique" around why it happens is nonsense. But if the attacking part of the story was interesting, the defense part was even more so, especially what's at stake with all this, which is nothing but regulatory capture of the industry.

Enjoying this piece? Then you'll love my newsletter, which explains AI in first principles and in words you can understand, for those allergic to hype but hungry for knowledge.

Join today.

Subscribe | TheWhiteBox by Nacho de Gregorio The newsletter to stay ahead of the curve in AI

The Response & Open-Source's Role

The interesting part of all this is that the incident response team from HuggingFace also used AI to investigate the AI-driven intrusion.

AIs versus AIs, that's the point we're reaching.

Its systems, supported by open models, specifically GLM-5.2, examined more than 17,000 recorded events to reconstruct the timeline and identify which credentials and infrastructure had been touched.

And what's the key takeaway? This event, which OpenAI is using to showcase how good its models are becoming, is in fact the perfect open-source marketing campaign.

Banning open-source would make this happen more

A typical argument made by closed Lab leaders and pushed by their lobbyists in Washington is that open-source models, the proliferation of this technology for free to everybody, will make such events more common, and therefore we should ban them.

Besides the obvious self-serving bias (this would instantly grant Labs almost infinite leverage over everyone and everything, forcing the entire US customer base to purchase only their products), this could make sense, right?

If OpenAI's models are already doing this shady stuff, why couldn't a Chinese model do the same six months from now?

And the answer is in today's story. It's precisely thanks to an open model, GLM-5.2, that HuggingFace could respond in time and principle, because the only way you can respond to swarm-level cyberattacks that AIs execute (thousands of different operations that require machine-level speed and attack surface) is with other AIs.

Much worse, although they tried, HuggingFace's cyber team could not use OpenAI's own models to defend from attacks from those same models, because guardrails prevented it.

Yes, the models OpenAI has could have served as a defense against this attack, but the AIs refused to help.

So the next obvious question would be: would you be comfortable knowing a small set of powerful individuals have cyber weapons at their disposal that nobody else has?

Weapons they can use to attack you, of which you can't defend yourself because the defense mechanism is banned? Because this is precisely the future Anthropic (and, it seems, OpenAI too) so desperately wants.

Such powerful technology must remain open

Banning open models is precisely the most dangerous decision Western nations could make, because the best way to diffuse the power of a certain technology is to make it widely available, not have a handful of billionaires hoard it.

Because what the closed AI Labs, massively unprofitable and desperate to keep their mystique enough to go public with all the bells and whistles, are doing is not fair game.

One thing is to create a business model so good that nobody can compete with you, with examples like Google. We can (and must) discuss antitrust issues if you get too powerful, but it's fair game to figure out and consolidate such an incredible business that beats everybody fair and square.

But another is having a business model so unprofitable that you need competition to be banned so that you survive. I don't care what your name is, OpenAI or Anthropic; if you can't make your business viable, you should fail, not be "saved" by the government by eliminating your competition.

HuggingFace's cyber team said it best: "Thanks to this approach [using a powerful open model], we were able to do in hours what would usually take days, and match the adversary's speed."

If anything, what looks like a great PR campaign for OpenAI "look how scary my models are" is in fact a PR campaign in favor of open source, as HuggingFace could not protect itself using frontier models because these models were guardrailed but used open models to counter that.

If this isn't the best advertisement one could possibly find for open models, I don't know what it is.

If anything, politicians should see this event and realize how vital it is that such a powerful technology, now a fully-fledged cyber weapon, doesn't fall only into the hands of the few.

I share similar thoughts in a more comprehensive and simplified manner on my LinkedIn (don't worry, no AI-generated content there either). As a reminder, you can also subscribe to my newsletter.