We Had a Hard Enough Time Stopping Computer Worms When They Couldn’t Think

I came across some research from Anthropic on self-replicating AI this week, and it got me thinking about something I hadn't really considered before. What happens when we start giving computer worms the ability to think for themselves?

Before I got into software entrepreneurship, my background was in information security and application security. And if you were working in IT or security in the early 2000s, you probably remember just how bad some of the worm outbreaks were.

Code Red, SQL Slammer, Blaster, Sasser. There were a bunch of them, and they caused some pretty serious problems.

For anyone who doesn't remember those days, these were relatively simple programs that exploited vulnerabilities in software, infected a machine, and then automatically started looking for other machines to infect.

They didn't have to trick someone into clicking a link. Once they found a vulnerable machine, they could move on their own.

What made them so disruptive wasn't necessarily what they did to the computers they infected. It was the enormous amount of network traffic they generated trying to spread.

I remember corporate networks becoming so overwhelmed that companies effectively stopped functioning. Internet connections were saturated. Internal networks were barely usable.

And one of the really frustrating things was that the network infrastructure you needed to fix the problem was the same infrastructure that was being overwhelmed by the worm.

Trying to push software updates, distribute patches, or even connect to the machines you needed to repair could become nearly impossible.

If that sounds exaggerated, it's worth looking back at what happened with SQL Slammer in January 2003.

According to research published by CAIDA, Slammer infected more than 90% of vulnerable computers on the internet in roughly ten minutes.

At its peak, infected machines were generating more than 55 million scans per second.

The disruption was significant. Bank of America's ATM network experienced problems. Airline flights were canceled. Emergency 911 systems were affected.

And the worm itself wasn't doing anything particularly sophisticated. It was just spreading as fast as it could.

That was 23 years ago.

Think about how much has changed since then. We didn't have smartphones as we know them. BlackBerry was pretty much the height of mobile business connectivity. Cloud computing wasn't running everything. Our homes, cars, appliances, and all sorts of other devices weren't constantly connected to the internet.

We were nowhere near as dependent on connected technology as we are today.

And we had an incredibly difficult time dealing with worms that were, by today's standards, pretty dumb.

Now imagine those worms could think

That's what caught my attention about Anthropic's research.

The researchers have been exploring whether AI models could replicate themselves across networks of relatively inexpensive devices, including things like Android TV boxes and other consumer hardware.

In one experiment, they managed to run a 26-billion-parameter model distributed across 15 inexpensive Android TV boxes.

They also tested whether AI agents could move through a controlled network, set themselves up on additional machines, and continue the replication process.

The broader concept is fascinating, and concerning.

Traditional computer worms generally had a fixed way of operating. They looked for a particular vulnerability, exploited it, and moved on.

If they encountered a computer that wasn't vulnerable, that was usually the end of the road for that particular attack.

But imagine an AI-powered worm that doesn't have to follow a predetermined set of instructions.

It encounters a computer, looks at what's running, figures out what vulnerabilities might exist, and decides how to attack it.

The next computer might be running completely different software, so it tries something different.

If an approach doesn't work, it can investigate why and try another.

And potentially, what it learns on one computer can help it compromise the next.

That's a pretty significant departure from the worms we dealt with 20 years ago.

In fact, Anthropic's researchers tested a version of this idea by allowing agents to inherit notes from previous replication attempts. The agents got better at completing the process over successive generations.

It's an early demonstration of how these systems could improve their effectiveness over time.

Instead of a worm that repeats the same attack millions of times, we're talking about one that could figure out how to attack each machine individually.

And then there's the vulnerability discovery problem

This is the part that makes the combination particularly concerning.

Earlier this year, Anthropic introduced Project Glasswing, which uses its Claude Mythos Preview model to identify security vulnerabilities in software.

The results were remarkable.

Anthropic reported that the model had discovered thousands of previously unknown vulnerabilities in widely used software, including operating systems and web browsers.

Some of those vulnerabilities had apparently been sitting there for decades.

One example was a 27-year-old vulnerability in OpenBSD, an operating system known for its focus on security.

Another involved a 16-year-old vulnerability in FFmpeg, a multimedia software library used in all sorts of applications.

These were vulnerabilities that had gone undetected despite years of code reviews, security research, and automated testing.

From a defensive security perspective, that's fantastic. AI could help us find and fix vulnerabilities that have been hiding in our software for decades.

But those capabilities aren't inherently defensive.

The same type of technology that helps a security researcher discover a vulnerability could potentially help an attacker discover one.

Now combine that with a self-replicating AI agent.

A traditional worm typically exploits a vulnerability its creator already knows about.

An AI-powered worm could potentially discover vulnerabilities as it spreads.

It encounters a machine, investigates the software running on it, identifies a weakness, and uses that weakness to compromise the system.

The next machine might require a completely different approach.

And the one after that, another.

These capabilities have been demonstrated separately, and bringing them together presents engineering challenges. But the direction is pretty clear.

I think it's inevitable that we'll eventually see self-propagating malware that uses AI to make decisions about how it spreads, whether someone deliberately builds it or it emerges from an autonomous system operating outside its intended boundaries.

How do we defend against that?

This is what concerns me most.

When SQL Slammer was spreading in 2003, the problem was enormous, but at least we understood what was happening.

It exploited a specific vulnerability. It generated recognizable network traffic. There were patches available.

The hard part was getting everything under control when the networks themselves were barely functioning.

Now imagine dealing with something capable of changing how it attacks systems as it goes.

You block one vulnerability, and it looks for another.

You protect one type of device, and it starts looking for weaknesses in something else.

You isolate part of your network, and it tries to find another way around.

And all of this could happen without a human attacker directing each step.

Our defenses are obviously much better than they were in 2003. We have sophisticated endpoint protection, behavioral detection, network segmentation, and automated response capabilities.

But those improvements don't eliminate the challenge of dealing with attacks that can adapt to the environment they're encountering.

And our dependence on connected infrastructure has grown enormously.

I think the only realistic answer is going to involve AI on the defensive side as well.

Systems that can identify unfamiliar attack behavior, investigate it, and coordinate responses at speeds human teams simply can't match.

That's already a direction the security industry is moving in. But the implications become much more significant when the threats themselves are autonomous.

We'll effectively have intelligent systems attacking and defending networks simultaneously.

And the speed and effectiveness of those defensive systems may determine how much damage a successful attack can cause.

This is a problem we're going to have to solve

I don't think the question is whether someone will eventually combine autonomous AI with self-propagating malware.

I think it's when.

The incentive is there. The underlying capabilities are developing rapidly. And the history of cybersecurity suggests that technologies capable of automating attacks eventually get used for exactly that purpose.

That doesn't mean the result will be some unstoppable worm that takes down the internet.

It does mean we should be thinking seriously about how to contain malware that can make its own decisions, discover new attack paths, and potentially learn from previous attempts.

Having lived through the worm outbreaks of the early 2000s, I remember how difficult it was to deal with software that did nothing more than exploit known vulnerabilities and replicate itself.

We had companies struggling to operate because their networks were overwhelmed. We had systems that couldn't be patched because the infrastructure needed to deploy those patches was barely functioning.

And that was in a world that depended on connected technology far less than we do today.

Now we're approaching a point where the software doing the attacking could potentially analyze the systems it encounters, discover vulnerabilities, and decide how to proceed.

That's a fundamentally different security problem.

I've spent much of my career building software, and I'm excited about what AI is making possible. But my background in information security makes it hard to ignore where some of these capabilities could lead.

We had a hard enough time stopping computer worms when they couldn't think.

Figuring out how to stop the ones that can is going to be one of the more important security challenges ahead of us.