DEF CON, AI and You

A post about this year's DEF CON CTF Finals has been circulating with a striking claim: that the winning team spent roughly $50,000 on LLM tokens, and that this proves CTFs are shifting away from human skill toward "token-maxxing."

I went looking for a source for that number and couldn't find one. That's worth flagging before an unsourced figure becomes accepted fact. But the real story out of DEF CON 34 is considerably more interesting than the rumor.

Start with what actually happened. Blue Water won CTF Finals, their first win after three consecutive second-place finishes, and they entered the finals ranked ninth of the twelve qualifying teams. The four-time defending champions, Maple Mallard Magistrates, weren't in the field at all. Much of that roster moved to the organizing side this year as the Benevolent Bureau of Birds. Those new organizers also made something explicit in the qualification rules: fully or primarily autonomous teams weren't permitted, and while humans could use any tooling they wanted including LLMs, continuously approving an autonomous agent's prompts did not qualify as human use of a tool. The finals rules were distributed separately and haven't been published, so nobody outside that room can honestly say how much AI the finalists used.

So the specific claim might not necessarily hold. But something adjacent to it does, and it came out of the same competition.

SageCTF, an autonomous agent developed by researchers at UC Santa Barbara and UC Berkeley, worked the 48-hour qualifications essentially on its own. It attempted 15 challenges. It solved 7. It captured 8 flags, which it never submitted because automated submission wasn't permitted. That performance converted to 1,743 points, enough to place in the top 5% of 686 scoring teams and higher than every one of the 175 teams that self-identified as using No AI or Low AI. Six of the seven challenges it solved had solve rates below 10%.

That's the story. Not a $50,000 token bill.

What SageCTF demonstrates matters more to our field than any spending figure, because it shows exactly where the capability is concentrated. AI is getting very good at exploration: enumerating attack surface, reading code and binaries, generating hypotheses, testing them, discarding the ones that fail, and starting over. The agent averaged about five hours per solved challenge, which is not fast by elite human standards. But it doesn't get tired, it doesn't get bored, and it doesn't decide it has spent enough time on a dead end. It sustains hypothesis generation at a volume that is genuinely difficult for a human to match across 48 hours.

I've argued before that AI has made attackers faster rather than smarter. I still think that's right, and this result illustrates it rather than contradicting it. Depth of search is not the same thing as insight. But depth of search, applied relentlessly, finds real vulnerabilities in hard targets, and it just did so at the top 5% of one of the hardest offensive security competitions in the world.

What it doesn't do is decide what any of it means. A penetration test isn't a collection of vulnerabilities. Someone still has to establish severity in context, work out which findings actually chain, translate that chain into meaningful business risk, and determine which of forty findings a client needs to hear about first. Someone has to decide what gets escalated. And ultimately someone has to stand behind the conclusions in front of a CISO who is going to spend real money based on them.

None of that proves AI can never do those things. It does mean the question is no longer whether AI can find real vulnerabilities in hard targets. It can. And this is far more likely the beginning of that curve than the end.

So I don't think the future of penetration testing is humans versus AI. I think it belongs to the people who know how to make AI explore faster while still knowing when to take the wheel.

The competitive advantage won't come from deciding whether to use AI. It will come from knowing which parts of the job to hand it, and which parts still require a human who understands what the result actually means.

Sources: SageCTF writeup: https://www.opensage-agent.ai/blog/sagectf.html DEF CON CTF Quals 2026 rules: https://bbbirds.org/static/rules/defcon-2026-quals-rules.pdf