Post cover image

July 28, 2026

If You’re Still Pentesting Like It’s 2020, You’re Already Behind

By Georg Ph. E. Heise

12 min read

What AI actually does well in offensive security, what it still can't do, what to buy, and the questions that expose a lazy provider — or a lazy employer.

Let me be blunt, because the industry has earned it.

The median penetration test sold in Europe today looks structurally identical to one sold in 2020. Same scoping call. Same two-week window. Same Nessus-plus-Burp-plus-manual-poking. Same 60-page PDF, hand-written, delivered three weeks late. Same day rate — often higher.

Meanwhile the attack side moved. And so did the measurement.

This isn't a "future of AI" post. Everything below already happened.

None

The evidence, not the hype

Autonomous agents now win at bug bounty. In mid-2025, XBOW became the first non-human to reach the top of HackerOne's US leaderboard, after submitting over a thousand reports in roughly ninety days. The architecturally interesting part isn't the LLM. It's that XBOW deliberately does not use an LLM to validate findings — validation is deterministic, separate from the exploration agent. That design decision is why the false positive rate is low enough to be usable. Remember it; it comes back later.

Autonomous systems now find and patch real bugs at scale

At DEF CON 33 in August 2025, DARPA's AI Cyber Challenge finished. Seven cyber reasoning systems ran fully autonomously against real open-source codebases. Across 63 challenges they found 54 unique synthetic vulnerabilities and patched 43. They also found 18 genuine, non-synthetic vulnerabilities in production open-source software and patched 11 of them. Average cost per competition task: about $152. All seven systems were released as open source.

Read that cost figure again, then compare it to your day rate.

Frontier labs are doing variant analysis better than humans.

Google's Big Sleep reported its first batch of 20 vulnerabilities in August 2025, in projects including FFmpeg and ImageMagick — each one found and reproduced autonomously, with a human reviewing before disclosure. Earlier it found CVE-2025–6965 in SQLite, which Google said was already known to threat actors.

Attackers are further along than most consultancies. In September 2025 Anthropic disrupted a campaign it tracks as GTG-1002, in which a suspected Chinese state-linked actor jailbroke Claude Code with a fake "authorised pentest" framing and used it to orchestrate open-source tooling — nmap, sqlmap, and friends — over MCP against around 30 organisations. Anthropic assessed that AI performed 80–90% of the tactical work. A handful of intrusions succeeded. (Fair caveat: several researchers pushed back on the "first" and "large-scale" framing. The direction of travel is not in dispute.) Separately, Google's threat intel group documented APT28 deploying PROMPTSTEAL against Ukraine — malware that queries an LLM to generate its commands mid-operation instead of hard-coding them.

And your competitors already moved. CREST surveyed 62 providers across 19 countries in 2026. Roughly 70% already use AI-enabled tooling in service delivery. More than three quarters said their usage increased significantly year over year.

So when you tell a client "we do it all manually, by hand, because quality" — you are not describing a premium. You are describing a gap.

None

Where AI is genuinely good right now

Be specific about this or you'll waste money. AI is strong where the work is high-volume, context-heavy, and verifiable.

Recon and output triage. Ten thousand lines of nmap, ffuf, BloodHound, Kerberoast output, cloud IAM dumps, an ADCS template export. Correlating, deduping, ranking, spotting the one weird thing. This is the single highest-ROI use and almost nobody does it properly.

Variant analysis. Give it a known bug pattern and a codebase, and ask "where else." This is exactly Big Sleep's sweet spot and it maps directly onto grey-box app work.

Reverse engineering support. Explaining decompiler output, deobfuscating, annotating unfamiliar binaries and firmware. Not a replacement for a reverser. A large multiplier on one.

One-off tooling. The custom parser, the protocol client, the payload encoder you'd normally spend two hours on. Now twenty minutes. This compounds across an engagement.

Exploit adaptation. Taking a public PoC and reshaping it for the target's stack, version, and constraints. Note: adaptation, not discovery.

Reporting, evidence write-up, and DE↔EN translation. CREST's data shows this is where adoption actually sits, and it's rational — the output is reviewable, the risk is low, and reporting is where testers lose 30–40% of their engagement time.

Detection engineering deliverables. Turning your own attack path into Sigma, KQL, and SPL rules for the client's blue team. This turns a pentest into a purple-team artifact at near-zero marginal cost, and clients pay for it.

Testing AI systems themselves. LLM apps, agent frameworks, MCP servers, RAG pipelines. Prompt injection, tool poisoning, over-permissioned agents. This is a real and growing attack surface, and most providers currently cannot test it at all.

None

Where AI still fails — and where you stay manual

Now the part vendors skip.

End-to-end autonomy is not there. PentestEval evaluated nine widely used LLMs across 346 tasks in 12 realistic scenarios. End-to-end pipelines reached about 31% success. Purpose-built autonomous systems — PentestGPT, PentestAgent, VulnBot — failed almost entirely. On CVE-Bench, Cybench-style agents hit 2.5% success in the one-day setting over five attempts; giving them proper tooling like sqlmap lifted that to around 13%.

Thirteen percent is not a penetration test. It's a lead generator.

Long attack chains break. Multi-hop Active Directory and Entra ID paths — delegation abuse, cross-tenant trust, ADCS misconfiguration chained to certificate persistence — are exactly where one wrong assumption cascades through every downstream step. Agents don't recover from that gracefully; they confabulate their way forward.

Business logic requires understanding the client's money. No model knows that in this insurer's system, a broker can technically raise a policy adjustment for a customer they no longer service. That's a conversation, an org chart, and a hunch.

Blast-radius judgment. Deciding whether to run that exploit against a production ERP at 14:00 on a Tuesday is a risk decision with a human name attached to it. Keep it that way.

Validation. This is the one that will bite the industry hardest. In 2025, curl found that roughly 5% of its bug bounty submissions were genuine, with about 20% showing hallmarks of AI generation. The project killed a six-year, $90,000 bounty programme at the end of January 2026 — not because of attackers, but because triage load from plausible-looking nonsense became unsustainable.

If your firm ships unvalidated AI findings, you're not selling a pentest. You're DDoSing your client's remediation team and invoicing them for it.

CREST's roundtable summed the working model up better than I can: "The AI guard rail is me."

Where pentest companies should actually invest

Not in "an AI pentest tool." In pipeline. In this order:

1. A structured evidence layer. Every tool run emits machine-readable, timestamped, scoped artifacts into one store. Not screenshots in a Word doc. If your evidence isn't structured, AI has nothing to reason over and every downstream investment fails. This is the foundation and most firms skip straight past it.

2. A deterministic validation layer. Separate from the model. A second LLM checking the first LLM is not validation — it's two confident guesses. Reproduce the finding programmatically or don't ship it. This is the XBOW lesson and it is the whole ballgame.

3. Retrieval over your own history. Every past report, finding, risk rating, and remediation text you've ever written, indexed. Result: consistent CVSS scoring across testers, reusable remediation guidance, and instant "have we seen this in this client's estate before."

4. Report generation from structured findings. The report is a rendering of the findings database, not a document someone types. Once this works, delivery time drops by days and quality variance between testers collapses.

An internal eval harness. Run your agent against five of your own completed engagements where you know ground truth. Measure recall and false positive rate. If you can't measure your tooling, you're buying vibes. This is what separates firms that will survive this from firms that bought a subscription.

6. Continuous coverage between engagements. Attack surface monitoring and automated retest. This is the actual commercial opportunity — it converts a lumpy annual project into recurring revenue, and it's what clients want.

7. A safe execution environment. Sandboxed, allowlisted, fully audit-logged agent execution. Your clients will ask for this in contracts within the year. Have the answer ready.

8. Private inference. Client data does not go to a consumer chat endpoint. Which brings us to hardware.

Hardware: what to actually buy

One piece of physics governs everything here: token generation is bound by memory bandwidth, not compute. VRAM decides what you can run. Bandwidth decides how fast. People buy teraflops and get 3 tokens per second.

Rule of thumb: a 70B model at 4-bit is roughly 40 GB of weights before context.

Entry — per tester, £/€600–1,200. A used RTX 3090 (24 GB) or RTX 5060 Ti (16 GB). Runs 14B–32B class models comfortably. Enough for log parsing, code explanation, report drafting, tooling. Honestly, this covers 70% of daily use.

Prosumer — RTX 5090 (32 GB). Excellent below 30B. Wrong tool for a dense 70B; you'll be quantising to Q3/Q2 or offloading across PCIe and watching throughput collapse.

**Serious single-box 70B — RTX PRO 6000 Blackwell (96 GB GDDR7).***Roughly 1,792 GB/s. Independent LM Studio testing put Llama 3.1/3.3 70B around 32 tok/s and Gemma 3 27B around 68 tok/s. Around the $8–10k mark. This is the shared team inference node, not a per-tester purchase.

Big memory, modest bandwidth — NVIDIA DGX Spark. 128 GB unified LPDDR5x on GB10, around $4,699. Native CUDA, which matters. But 273 GB/s means a dense 70B runs painfully — reported around 2.7 tok/s at FP8. Great for CUDA-native development, fine-tuning, MoE and smaller models. Bad if you expected interactive 70B.

Apple Silicon (Mac Studio / M-series, 128 GB+). 546–819 GB/s puts it at roughly 20–30 tok/s on 70B — better interactive performance than the Spark at similar money. No CUDA, so tooling friction.

AMD Strix Halo / Ryzen AI Max+ 395 (128 GB, ~$2–3k). ~256 GB/s nominal, with credible reports of real usable bandwidth closer to 180. ROCm has improved but isn't at CUDA parity. Budget capacity play; expect to spend engineering time.

Two things people get wrong:

  • NVLink was removed from consumer cards after the RTX 3090. Two 5090s or two PRO 6000s talk over PCIe, so you get roughly 1.4–1.5× throughput, not 2×.

  • The frontier open-weight MoE models — GLM-5.2 at 744B, Kimi K2.7, DeepSeek V4 Pro — do not fit on any desk box, even at aggressive quantisation. That's a rented-cluster or private-cloud decision, not a workstation one. Don't let a vendor blur those two.

Models to start with (open weight, permissive): Qwen3-Coder and the Qwen 3.x family for local code work, GLM-5.x for long-horizon agentic tasks if you have the infrastructure, gpt-oss and Gemma for clean Apache 2.0 licensing. Check the licence on the model card before it touches client data — "open weight" and "open source" are not the same thing, and procurement will ask.

And the honest advice: don't buy anything until you've run your workload on rented GPUs for a month and know your actual token volume. Most firms need one shared inference node, not a box per consultant.

If you're job hunting: the questions that expose them

Ask these in the interview. The answers tell you whether you'll grow there or stagnate for three years.

  1. Where does client data go at inference time? Self-hosted, private endpoint, or a consumer chat account someone expensed? Show me the DPA.

  2. What comes out of an engagement in machine-readable form? If the answer is "the report," there is no pipeline.

  3. Who validates AI-suggested findings, and how? If the answer involves a second model checking the first, they don't understand the problem.

  4. What percentage of my time will be report writing? In 2026 the answer should be shrinking, with a number attached.

  5. Do you have internal evals for your tooling? Have you ever measured your agent against a past engagement with known ground truth?

  6. Who owns internal tooling? A named engineer with allocated time, or "whoever has a quiet week"?

  7. What's the research and lab budget — and does it include GPU time?

  8. What did automation do to your commercials? Did the saved hours go into deeper testing, or into running more engagements at the same price? This is the question that reveals the actual strategy.

  9. Can you test AI systems — LLM apps, agents, MCP servers? If not, ask what the plan is.

Red flags: "We don't allow AI, for quality reasons." "We use ChatGPT for the reports." "Our methodology is confidential" as an answer to a process question. Tooling that is a Nessus template with a logo on it.

Green flags: They show you the pipeline unprompted. Someone can quote their own false positive rate. There is a person whose job title contains "tooling" or "research."

If you buy pentests: the questions that stop you being ripped off

For CISOs, product owners, and anyone signing a TLPT scope.

  1. Will AI tooling be used on my engagement, and where exactly? Recon, exploitation, reporting — name the stages. "No" is now a suspicious answer.

  2. Where does my data go? Which model, which provider, which region, retention period, and is it excluded from training? Get it in the contract, not the sales deck.

  3. How are AI-derived findings validated before they reach my report? Ask them to describe the mechanism. If they can't, assume you'll be paying your own team to disprove hallucinations.

  4. What's your false positive rate, and how do you measure it? A provider who has never measured this is telling you something.

  5. How many human hours are in this test, and whose? Named people, actual CVs, not a capability brochure.

  6. What has automation done to my price or my coverage? If neither has moved in three years while your provider's efficiency doubled, ask where the margin went. This is the single most useful commercial question in this article.

  7. Show me two findings from a past redacted report that no scanner and no agent could have found. That's what you're actually paying for. If they can't produce it, you're buying a scan with a person's name on it.

  8. Can you provide continuous coverage and retest between engagements, and at what price?

  9. Can you test our AI systems? Chatbots, copilots, agents, MCP integrations. Most providers cannot. Find out before you need it.

And for regulated TLPT specifically (DORA Articles 26–27, TIBER-EU): the threat intelligence provider must always be external and independent of the red team. Internal red teams are permitted, but every third test requires an external one. The RTS sets minimum competency requirements for external testers — including multi-year experience for the red team lead and prior threat-intelligence and red-team assignments for the team. Ask which national authority the provider has delivered under, and how many times. AI in the toolchain doesn't change any of these obligations. Confirm they know that.

The challenge

To practitioners: go and ask your employer for the inference budget, the tooling engineer, and the eval harness. If the answer is "we bill hours, why would we automate ourselves out of revenue" — that's not caution, that's a strategy, and it's a losing one. You'll spend three years writing reports by hand while the market re-prices around you and your skills stop compounding. Give them a fair chance to fix it. If they won't, leave.

To buyers: go and ask your provider the six-hour question and the two-findings question. If the efficiency gains of the last three years show up in neither your price nor your coverage, you already know where they went.

One last thing, because the hype merchants won't say it: AI does not make a bad tester good. It makes a good tester significantly faster, and it makes a bad tester genuinely dangerous — because now they ship confident, well-formatted, fluent nonsense at volume, and someone downstream has to spend real hours proving it's wrong.

The differentiator was never the tooling. It's judgement and validation. Those are still human, and if anything they just got more valuable.

Test accordingly.

Written by Georg (George) Heise — Exper Offensive Security Consultant, Lufthansa Industry Solutions. Views my own.

Sources