Sol in the shade: benchmarking Opus 4.6 and GPT-5.6 Sol for finding zero-days

I pointed Opus 4.6 and GPT-5.6 Sol at a big, hardened open source project and told them to hunt for zero-days. Opus 4.6 out-counted GPT-5.6 Sol on the raw numbers, but the thing that actually surprised me is how completely different the two results turned out to be.

TL;DR

• One went deep, one went wide, and wide won the count. GPT-5.6 Sol drilled a single high-value seam. Opus 4.6 swept a much larger surface and came back with more.
• The harness matters. I believe it's a critical part of how these models hunt vulnerabilities. Each model has its own way of thinking, and what really matters is finding the model-harness pair that fits.
• The Great GPT-5.6 Sol lost this round. By the raw stats it found only 15 Findings, while Opus found 20. Opus also logged 2 valid criticals to Sol's zero.
• One billion tokens is too small to show what these models can really do. I believe both of them have a lot more in the tank, and I'm going to keep pushing this research further.
• Running both is the real win. Out of all the findings, only one was literally the same bug. Merging the two runs roughly doubles the ground a single run covers.

Hello, my name is Teguh. I'm a developer and a security researcher, and lately I've been building a personal project called Cleave, a harness for security research and bug bounty hunting. I know there are already a lot of open source security harnesses out there (Shannon, RedGPT, Strix, and so on). I studied all of them, I mean I literally dissected them, and none of them fit the way I actually hunt bugs. So I built my own.

The idea behind Cleave is that security research isn't a cyclic process. It's a hill climb, you collect evidence and gadgets along the way and use them to build up a full exploitation chain, a real kill chain. I also built Cleave for both agents and humans, so it works the way I do. It takes notes like a human, it writes like a human, and that means a human and an agent can work the same case together.

So I wanted to test my harness against a hardened open source target, a big one, using Opus 4.6 and GPT-5.6 Sol. I capped each model at roughly 1 billion tokens. I soft-stopped the harness once it hit that cap, letting each one finish whatever exploitation or progress it had in flight. So the real spend for both landed a bit over 1 billion, somewhere under 1,100M tokens for each.

it's August, why 4.6 bro, are you nuts?

my bad, but Anthropic revoked my CVP for no reason T-T, so I can't touch Opus 5. I'd love to try it the moment I get my CVP back

So, here's what I got.

How the run worked

The target was a big open source enterprise platform, the kind of thing that sits in the middle of a company and runs both its APIs and its internal plumbing for a whole crowd of customer organizations at once. I left it in the hardened setup its own maintainers recommend for production (its required for valid finding to be a CVE), so nothing here rides on some sloppy default that a careful operator would have closed already. And of course i run it in different identical environment.

One caveat up front. Each model ran once. This is a single snapshot, not an average, and a second pass of either one would land somewhere a little different. Aikido saw the same thing when they repeated runs of a single model, and it clearly matters here too.

The headline numbers

On the raw count the two runs look alike in method but split hard in shape. GPT-5.6 Sol turned 44 Leads into 15 Findings. Opus 4.6 turned 64 into 20. Both promoted about a third of what they opened, 34 percent for Sol against 36 for Opus, so neither one was actually better at closing a Lead once it had one. Opus just opened more of them, ran a slightly bigger crew of 75 agents to Sol's 69.

Where they split hardest is how loudly they rate what they find. Sol never once called anything critical and only reached high a single time, parking most of its work in the medium band. Opus reached for the top of the scale freely, with three criticals, seven highs, and a clean ten at its peak against Sol's ceiling of 8.1. Score for score, Opus sits more than a full point higher. Whether that's sharper judgement or just a heavier hand is the open question, and I'll come back to it.

Where they agreed, and where they scattered

Bucket every Finding by the weakness underneath it and the split explains itself. Six buckets show up on both sides, and all six tell the same story, access control and identity, one customer reaching another customer's data or acting with authority it was never granted. That is where this platform's real risk lives, and both models walked straight to it. Past that shared core they scatter. GPT-5.6 Sol alone cracked a header-parsing quirk in the gateway and a browser-driven request forgery, two fiddly corners that only pay off if you sit with them. Claude alone opened a much wider fan, with XXE flaws, server-side request forgery on three separate surfaces, a race, and an entire second product bundled in the same download that GPT-5.6 Sol never touched. Together the two runs hit twelve weakness types.

The overlap is thinner than it looks. Twelve times the models landed on the same class of weakness, and exactly once on the very same bug, an export feature that handed a low-privileged user someone else's live API credentials. Every other match was the same idea reached through a different vector.

The Severities

That same temperament runs straight through how each model rates danger. GPT-5.6 Sol is the cautious one and it under-claims. Several of its cross-tenant write findings, where an attacker sitting in one organization rewrites or deletes another's data, sit at medium when the impact it actually showed earns a high. The upside is it never cries wolf. The downside is that some genuinely nasty work reads as routine. Claude uses the whole range and nails the ends, a real critical for an internal service left open to anyone across every tenant, and a correct low for an administrator abusing the tenant it already owns. The price of that range is a steady one-notch lift through the middle, including a high for the shared export bug that GPT-5.6 Sol rated a medium. Both, to their credit, price on impact they reproduced rather than impact they described, and both wrote down the things they tried and could not land.

A few of Claude's louder calls come with an asterisk, and most of its write-ups already carry one. The top critical stands only if the hardened build really leaves that internal service exposed, and it falls the moment an operator puts a proxy or an access rule in front of it. Two more lean on treating a bundled companion product as current and on a privilege gap a stricter role policy would close. That does not make them wrong. It makes them conditional, and a benchmark against a hardened, current deployment is exactly where the conditions belong in writing.

What it means

The lazy read is to crown a winner. Claude-Opus-4.6 takes the scoreboard, and the useful read is that the two models are complementary. Their findings barely touched, so a team that ran both and merged the output walks away with far more than the better single run gave it. Aikido found the same shape when it pooled repeated runs of one model, that coverage comes from more attempts rather than a bigger single pass, and it holds here across two different models instead of three runs of one. Depth and breadth are dials, not verdicts. The deep run returns fewer, better-proven findings on one theme. The wide run returns more variety and reaches what the other never opened. The strongest result came from having both in the room.

Where this goes next

I don't think either run hit its ceiling, so the next round gets a much longer time horizon and a bigger token budget, and I expect that alone to uncover more bugs out of both of them. The version I really want to try is one agent driving both models at once, letting Sol and Opus argue over the same Lead in real time instead of running in separate rooms, so I get the depth and the breadth in a single pass. Once I report these upstream and go through the CVE process I'll come back with the honest tally of valid CVEs, duplicates, and rejects, whichever way it falls.

Till then, lets break more stuff. See ya!