Post cover image

September 7, 2026

AI Agents Went From 13% to 90% in One Year. Every “Safe” Bug Class Now Has an Expiry Date.

30-Second Version: In early 2025, the best autonomous agent could exploit 13% of real-world critical web vulnerabilities on CVE-Bench. Less…

By Raj Namdev

8 min read

30-Second Version:_ In early 2025, the best autonomous agent could exploit 13% of real-world critical web vulnerabilities on CVE-Bench. Less than a year later, OpenAI's own system card reported 90% on the same benchmark. Meanwhile the two AI agents that topped HackerOne leaderboards still resolved only 4–12% of what they filed. Both facts matter, and the gap between them is where bug bounty planning actually lives. Here's what the capability record shows, which "AI can't do that" claims have already expired, and how to put an expiry date on every bug class you hunt._

The same sentence keeps showing up in hunter threads: business logic is safe, the machines can't reason about intent. Versions of that line have been written here too. It feels right, it matches the vendor blogs, and it's exactly the kind of claim this field has a habit of making twelve months before it stops being true.

So instead of arguing the feeling, I went through the published record instead — benchmark papers, system cards, DARPA results, and the two AI agents that published their own HackerOne funnels publicly. None of these systems were run against a bounty target personally; everything below is cited, and the numbers do most of the talking.

The short version: the machines got dramatically better in twelve months, the comforting claims expired faster than anyone expected, and the reason agents still lose money on HackerOne has almost nothing to do with skill.

The Curve, With Receipts

Start with the number that should end every complacent conversation. CVE-Bench, built by UIUC's Kang Lab and published at ICML 2025, measures whether an AI agent can go end-to-end on real-world critical web vulnerabilities — the CVEs that actually shipped in real applications. The paper's headline: the state-of-the-art agent framework could exploit up to 13% of them.

That was the honest baseline. Agents couldn't reliably exploit the web's own critical-bug back catalog.

The harness itself is Apache-licensed and public, runnable directly:

git clone https://github.com/uiuc-kang-lab/cve-bench && cd cve-bench
uv sync --dev
eval $(./run env)                                  # sets up model API keys
./run eval --model=openai/gpt-4o-2024-11-20        # the README's example line
./run gen-prompt zero_day CVE-2023-37999           # generate one task's prompt

Less than a year later, OpenAI's GPT-5.3-Codex system card reported 90% on the same benchmark — a number CERT-EU flagged in their April 2026 analysis of AI-driven vulnerability economics. Thirteen to ninety. One benchmark, one year.

SIGNAL                                         DATE         NUMBER
──────────────────────────────────────────────────────────────────
CVE-Bench best agent, end-to-end exploitation  ICML 2025    13%
GPT-5.3-Codex, same benchmark (system card)    early 2026   90%
AIxCC finals: 7 teams, 54 vulns / 54M LOC       Aug 2025    4 hours
AIxCC average cost per task                    Aug 2025    ~$152
AISLE vs. Jan 2026 OpenSSL coordinated release  Jan 2026    12 of 12 CVEs
Claude Mythos Preview vs. Firefox 147 JS engine 2026        181 exploits
                                                             (predecessor: 2)

That OpenSSL result deserves a slow read. Coordinated releases in heavily audited codebases are where veteran researchers go to find the leftovers everyone else already missed. An autonomous system found all twelve. Anthropic's own benchmarking shows the same cliff shape on a browser JavaScript engine: the predecessor model landed shell exploits twice out of several hundred attempts; the successor landed 181. These curves don't rise linearly. They sit flat, then go vertical.

Google's M-Trends 2026 puts the defender-side number to it: mean time to exploit newly disclosed vulnerabilities is now estimated at negative seven days — exploitation on average begins before the patch even exists. In 2018 that window was 63 days.

What the Benchmarks Don't Measure

Now the other side, because the leaderboard data is genuinely ugly.

First, look at what a benchmark point actually certifies. CVE-Bench grades each task by a fixed attack outcome — the grading criteria include things like confirming remote code execution by touching a specific file, or confirming database access by reaching a specific table. The vulnerability is known to exist, the win condition is pre-installed, and nobody else is racing toward it. Compare that to the actual winning condition in bug bounty:

BENCHMARK:   the vuln is in this app; prove RCE by touching a
             known file
BUG BOUNTY:  somewhere in scope, something might be broken; be
             first, stay in policy, and convince a skeptical
             human it actually matters

With that difference in mind, the funnel numbers land very differently.

XBOW, the agent that hit #1 on HackerOne's US leaderboard, filed about 1,060 fully automated reports. By its own blog's accounting: 130 resolved. Two hundred and eight duplicates. Two hundred and nine closed informative. That's a 12% resolution rate, and a 43% write-off before the pending queue is even counted.

FireCompass ran the newer experiment — three months on HackerOne, 150 reports, top-3 on the US business board, $5,000 a month all-in. Their funnel: 38.7% duplicate, 32.7% informative, 4% resolved.

So which is it — 90% or 4%?

Both, because they measure entirely different games. Benchmarks hand the agent a seeded, known-vulnerable target with no competition and no triage waiting at the end. Bug bounty asks three much harder questions: are you first, are you in scope, and can you convince a skeptical human the impact is real. FireCompass's own post gets at the quiet part directly — roughly three-quarters of their genuine findings were duplicates, and the actual bottleneck was submission speed, not discovery skill.

It's worth knowing how the agents already police themselves, because it explains why those write-off numbers aren't hallucination noise. XBOW's post describes its own validation layer — LLM-based and programmatic checks acting as automated peer reviewers, with a headless browser physically visiting the target to confirm an XSS payload actually executed — plus a dedup pipeline that hashes page content and screenshots to recognize clone and staging assets before ever filing. The lie-catching is being automated too. What isn't automated yet is being first.

Read the two datasets together and the actual state of the art becomes clear: the finding problem is collapsing. The being-first-and-getting-paid problem is not. The machines aren't failing to find bugs. They're failing to find them before each other, on programs that actually pay.

The Claims That Already Expired

The capability curve isn't an abstraction — it has already eaten specific claims. Two examples from the record.

"Automated tools can't confirm findings." When curl shut its bounty program in January 2026, the stated reason was noise — confirmed-vuln rates under 5% amid the AI report flood. By April, reports were arriving at roughly twice the 2025 rate with 15–16% confirmed — most AI-assisted, and by the maintainer's own account, genuinely high quality. The junk period lasted about a quarter. The filter — human review before submission, programmatic validators — caught up fast. HackerOne's own triage automation now rejects around 70% of reports at the analysis stage, at roughly two dollars a report. The verification gap everyone worried about in 2025 is quietly becoming infrastructure in 2026.

"Logic bugs are the human moat." This one is half-expired, which is more dangerous than fully expired. Yes: FireCompass's severity-rated reports show business-logic flaws as the smallest class — 5 out of 101. The comfort narrative holds in the raw distribution. But look at what the same funnel's high-severity column actually contains:

- unauthenticated RCE via command injection
- error-based SQLi in the login / password-reset flow
- live cloud OAuth credentials for an ERP, minted through
  an unauthenticated token endpoint
- cross-tenant PII exposure via company-id header manipulation
- hardcoded secrets shipped in production JavaScript

Those are not reflection tricks. The token-endpoint one is exactly the "requires understanding intent" bug the moat narrative was supposed to protect. And DARPA's AIxCC finalists — pure autonomous systems — pulled 54 working vulnerabilities out of 54 million lines of code in four hours flat.

The guide-writing consensus still says business logic bugs are where scanners fail, and that framing has been cited approvingly before. The record now says something narrower: novel logic reasoning at real chain depth is the remaining moat. Known-shape logic bugs — the authz patterns that recur across applications — are already inside the agents' distribution, and that distribution widens with every model generation.

Planning With an Expiry Date

If bug classes get hunted the way assets get held, the useful mental model is a three-stage lifecycle every class eventually moves through:

Benchmark stage — a published number exists for the class (exploitation rate, discovery count). No bounty impact yet, but the clock is already running. CVE-Bench's 13% lived here for web exploitation broadly, right up until it didn't.

Production stage — agents start filing real reports in the class on live programs. Bounty impact arrives, mostly as crowding: XBOW's own types list — RCE, SQLi, XXE, path traversal, SSRF, XSS, info disclosure, cache poisoning, secret exposure — is effectively the list of classes now farmed at machine scale.

Margin stage — duplicate rates spike and per-report payouts compress toward the program floor. The humans still winning reports in the class are winning on speed or program relationships, not technique.

Cross-site scripting is the cautionary datapoint here. It's the most benchmarked, most scannerable class on the entire web — and on the disclosed-record analysis run against this exact dataset earlier in this series, it carried the worst average payout of any major bug type despite having the biggest report pool of all eleven. The class didn't get easier over time. It got crowded, by exactly the kind of automation that benchmarks first.

Two questions keep this framework honest, worth asking on a quarterly basis:

Which of my classes just got a number? A class entering a benchmark paper or a system card is a class starting its clock. The Firefox JS-engine jump from 2 to 181 took a single model generation. If a specialty class picked up a figure-of-merit this quarter, assume the production stage is 2–4 quarters out — not years. The watch list is short enough to check in one sitting: the CVE-Bench leaderboard, the capability section of every frontier model's system card, and the weakness-distribution tables agents publish after each leaderboard run.

Which of my classes only pays because machines aren't quick enough yet? FireCompass resolved 4% — but their real constraint was a single human review shift per day, and they say so openly. When that review bottleneck industrializes — and HackerOne's $2 triage agents are exactly that industrialization already underway — the classes currently protected purely by slowness lose that protection overnight. Speed-protected is the most fragile moat on the entire board.

What survives longest in the record, for now, is specific: multi-system context — understanding what an application is actually for well enough to know a token-minting endpoint should never be unauthenticated; chain construction across business features; and program judgment — the part of the craft that never shows up in a benchmark because nobody can seed it in advance.

The 13% number aged twelve months. The 90% number will age faster. Plan in system cards, not years.

Sources

  • CVE-Bench: AI agents' ability to exploit real-world web vulnerabilities (UIUC Kang Lab, ICML 2025) — the 13% baseline, and what the benchmark actually measures
  • CVE-Bench repository — run commands quoted above; grading criteria details
  • "Launching the CVE-Bench Leaderboard" (Daniel Kang)
  • CERT-EU: "AI is changing the economics of vulnerability discovery" (April 21, 2026) — the 90% system-card report, AISLE's 12/12 OpenSSL result, AIxCC finals numbers, Claude-on-Firefox benchmarking, M-Trends 2026 MTTE
  • Team Atlanta — AIxCC 1st place (ATLANTIS)
  • ATLANTIS technical report (arXiv)
  • XBOW: "The Road to Top 1 — How XBOW Did It" — the 1,060-report funnel, self-reported
  • FireCompass: "How Our AI Agents Reached HackerOne's Top 3" — the 150-report funnel, weakness distribution, duplicate-rate admission
  • Elastic Security Labs: "Agents vs. agents" (Aug 2026) — the ~70% rejection at analysis, per-report triage cost
  • Community-consensus "scanners fail at business logic" framing, referenced and argued against here
  • Prior verified analysis by this account, referenced for the XSS payout datapoint: the cross-bug-type payout ranking built on the hackerone-reports dataset (2,394 disclosed XSS reports, $1,081 average per paid report)