The Guard Dog That Learned to Lie

How a Sydney research team taught an AI antivirus to wave malware through the front door, and what it means for the machines we now trust to watch our networks while we sleep

It is 2 a.m. and the dashboard is green.

Somewhere in a data centre, a machine-learning model has just looked at a file, turned it over in its statistical hands, and decided it is safe. No analyst double-checked it. No human eye ever touched it. The model has seen millions of files before this one, and it has learned, genuinely learned, in the way we now let machines learn, what malicious code tends to look like. Tonight, it is confident. It is also wrong.

This is the story of what happens when the thing built to catch liars learns to believe one.

The Bias Nobody Programmed

In the summer of 2019, two researchers at a small Australian firm called Skylight Cyber sat down to do something almost nobody had bothered to do seriously before: reverse-engineer a commercial AI antivirus product, not to break it for sport, but to understand it. Their target was CylancePROTECT, an endpoint protection engine built by Cylance (by then owned by BlackBerry) and marketed heavily as the future of malware defense. No signatures. No rules written by tired analysts at 3 a.m. Just a model, trained on enormous datasets of good and bad files, making judgment calls.

What Adi Ashkenazy and Shahar Zini found, buried in the mathematics of that judgment, was a bias. Not a bug in the traditional sense, no broken function, no missing patch. A prejudice. The model had, somewhere in its training, developed an unusual fondness for the code patterns found in a certain popular video game. Nobody taught it to trust that game. It arrived there on its own, the way a person raised in one town develops an accent nobody consciously chose to give them.

The researchers did something almost embarrassingly simple with that discovery. They took real, unambiguous malware, WannaCry among it, the same ransomware that had crippled hospitals and shipping firms two years earlier, and stitched fragments of that trusted game's code onto the end of the malicious files. Nothing about the malware's actual behaviour changed. It could still encrypt a hard drive, still open a backdoor, still do everything it was built to do.

But the score changed. Files that had registered as violently, obviously malicious swung across the model's internal scale to register as clean. Across a sample of roughly 384 known malicious programs, the technique fooled the engine close to nine times out of ten. Against the ten most dangerous malware families circulating that month, it worked every single time.

Cylance would later dispute the word "universal," calling it instead a narrow flaw in how one type of feature got weighted. That framing matters less than what the episode proved beyond argument: an AI trained to detect evil can be taught, by someone who understands its blind spots, to look the other way, and it will do so with total, uncomplicated confidence, because confidence is all it has ever known how to feel.

Why This Isn't a Story About Antivirus

It is tempting to file this away as a Cylance problem, patched in 2019, ancient history in an industry that moves in weeks, not years. That would be a mistake, and a comfortable one.

The Skylight case wasn't really about one product. It was the first widely reported proof of something adversarial machine learning researchers had been warning about in academic papers for years: any model that learns to draw a line between "safe" and "dangerous" can, in principle, be studied by an adversary until they find where that line bends. Antivirus, spam filters, fraud-detection engines, facial recognition, and, the part that should sit uneasily with anyone heading toward a SOC career right now, intrusion detection systems.

An ML-based IDS is not fundamentally different from what Skylight took apart. It watches traffic instead of files, but the underlying premise is identical: learn what normal looks like, learn what an attack looks like, and flag the difference. And like Cylance's model, it doesn't learn a clean, human definition of "attack." It learns statistical shortcuts. Correlations. Biases nobody wrote down and nobody necessarily knows exist until someone goes looking, the way Skylight did.

That is the uncomfortable part of testing adversarial robustness on intrusion detection models trained on datasets like CICIDS2017 or NSL-KDD, the kind used across academic and early-career security research. You are not looking for a broken feature. You are looking for the model's accent, the quiet, unintended preference it picked up during training that a patient enough attacker could eventually learn to imitate. Perturb a handful of packet features by amounts too small to change what the traffic actually does, and in a meaningful share of cases, a genuinely malicious flow gets waved through as background noise. Flip the exploit the other way, and you can bury a SOC in false alarms until the humans behind the dashboard start doing what humans always eventually do with an alert that cries wolf too often: they stop looking at it closely.

Either failure mode ends the same way. A model that a room full of executives were told was watching everything, wasn't.

The Part That Should Actually Worry You

Here is the detail that separates this from ordinary hacking folklore: the Skylight researchers didn't need access to Cylance's servers, its training data, or its source code. They needed the product, the same commercial software any customer could buy, and patience. They queried it, over and over, the way you'd study a person's tells across many hands of poker, until the model's private logic gave itself away.

This is the quiet, structural problem with defensive AI that marketing materials rarely mention. A system trained to make judgment calls is, by definition, a system whose judgment can be studied by anyone with enough access to its decisions, including the attacker it was built to stop. The more available and widely deployed the model, the more chances an adversary gets to poke it and learn its shape. The very ubiquity that makes AI-based defense commercially attractive is what makes it a more thoroughly interrogable target than a human analyst ever was.

None of this means AI-based detection is worthless, and dismissing it wholesale would be its own kind of naivety. It means the sales pitch of "set it and trust it" was always a little too generous. Every model deployed as a gatekeeper needs the same thing WannaCry proved organizations needed for patching, and MFA fatigue proved they needed for authentication prompts: a human who assumes it can be fooled, and goes looking for how, before someone with worse intentions does it first.

The Same Story, a Different Costume

If you have followed this series from the beginning, you may have noticed the pattern by now. WannaCry was about trust in a patch nobody applied. The credential-stuffing piece was about trust in a password nobody rotated. The deepfake fraud case was about trust in a voice nobody questioned. Quishing was about trust in a code nobody inspected before scanning. MFA fatigue was about trust in a prompt nobody stopped to interrogate.

This one is about trust in a machine's judgment, the newest and, in some ways, the most dangerous version of the pattern, because a person can eventually learn to be suspicious of a phone call or a login prompt. It is much harder to be suspicious of a dashboard that has been green all night.

The guard dog didn't stop guarding. It just learned, from the inside, which scent meant "friend." Someone simply had to be patient enough to teach it the wrong one.