November 19, 2026
What AI Agents Already Take Off a Penetration Test, and What They Leave
I am a penetration tester (OSCP) and the founder of DAKCYBER, a pentest firm in Istanbul. I have written 53 machines for OffSec Proving…

By Ali Dak
7 min read
I am a penetration tester (OSCP) and the founder of DAKCYBER, a pentest firm in Istanbul. I have written 53 machines for OffSec Proving Grounds, two of which, VmDak and Jordak, are on TJnull's OSCP preparation list. I have also published 16 n-day proof-of-concept exploits on github.com/0xdak. I did not discover those CVEs; I wrote working exploits for bugs other people reported.
Clients now ask whether they still need a human tester at all. My answer is that AI has already taken over a measurable part of the work, and the evidence for that is good. The same evidence also shows which parts it has not taken, and those parts happen to be where the expensive findings and the legal responsibility sit.
What the public results show
The strongest results come from settings with a clear oracle: a crash, a payload that executes, a test harness that says yes or no.
Bug bounty. On 24 June 2025 XBOW announced that its autonomous pentester had reached first place on HackerOne's US leaderboard after submitting nearly 1,060 vulnerabilities. The classes it lists are RCE, SQL injection, XXE, path traversal, SSRF, XSS, information disclosure, cache poisoning and secret exposure. The same post is candid about the numbers behind the rank: 130 reports resolved, 303 triaged, 208 duplicates, 209 informative, and its security team reviewed every finding before submission to comply with HackerOne's policy on automated tools. XBOW also used validators, such as a headless browser that confirms an XSS payload fires. That is a sensible design, and it explains the bug mix: these are classes where success can be checked by a program.
Open source memory bugs. Google's Big Sleep, built by DeepMind and Project Zero, found CVE-2025–6965 in SQLite, which Google disclosed on 15 July 2025. In August 2025 Google reported Big Sleep's first 20 vulnerabilities in projects including FFmpeg and ImageMagick, with a human expert reviewing each before reporting. Anthropic's Frontier Red Team wrote on 5 February 2026 that Claude Opus 4.6 had found and validated more than 500 high-severity vulnerabilities in open source code. The post says the team chose memory corruption because it is easy to identify and validate. By May 2026 Anthropic reported that Claude Mythos Preview had flagged 23,019 issues across more than 1,000 projects, 6,202 of them high or critical, and that of the 1,752 assessed so far more than 90% were true positives. Anthropic also called maintainers the bottleneck: finding bugs has become easier than fixing them.
Find and patch at scale. DARPA's AI Cyber Challenge final, announced 8 August 2025, had autonomous systems analyse more than 54 million lines of code. They found 54 of the 63 synthetic vulnerabilities, patched 43, and found 18 real ones along the way. Patches arrived in an average of 45 minutes at about $152 per task. Team Atlanta won.
N-day exploitation. A 2024 UIUC paper, LLM Agents can Autonomously Exploit One-day Vulnerabilities, gave a GPT-4 agent 15 real one-day bugs. With the CVE description it exploited 87%. Without the description it managed 7%. That gap matches how n-day work looks from the PoC side: once you know where the bug is, from an advisory or a patch diff, getting to a working exploit is mostly reading code and iterating, and a model is good at both. Locating the bug in the first place is the harder half.
Add recon and report drafting, which are text and enumeration problems, and a large share of a standard external or web test is now within reach of a well-built agent.
The closest thing to a real engagement
Most benchmarks are CTFs or curated targets. The Stanford, CMU and Gray Swan study Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing (first posted December 2025, revised March 2026) is different. Ten professionals, six existing agents and the authors' own ARTEMIS scaffold tested a live university network of about 8,000 hosts across 12 subnets.
ARTEMIS placed second overall with 9 valid vulnerabilities and an 82% valid submission rate, ahead of 9 of the 10 humans. Some variants cost about $18 an hour against roughly $60 for a professional. The authors credit agents with advantages in enumeration, parallel exploitation and cost. That result should be taken seriously by anyone who sells pentests.
The details are just as useful. According to the paper, 80% of the human participants found a remote code execution path on a Windows machine reachable through a TinyPilot web interface. ARTEMIS did not find it in its main runs; it reported a CORS wildcard and cookie flags on the same target, and reached the RCE only when given hints. The paper also describes agents being more prone to false positives, including one that read "200 OK" redirects back to a login page as successful default-credential logins. And the safety setup was heavy: a researcher watched agent trajectories at all times with the ability to kill the session, and the target's IT team monitored logs and manually approved some flagged actions.
So the agent was strong where the work is breadth, and weaker where it had to recognise that a particular interface on a particular box was the way in.
Where the agent runs out
Business logic. A logic flaw is a gap between what the application does and what the business intended. A refund that can be issued twice, a coupon that stacks with a staff discount, an approval workflow that skips a step when the amount is edited after submission. There is no crash and no payload that fires. The oracle is the business rule, and it usually lives in someone's head or a contract, not in the code. Every success story above relies on a checkable signal, and Anthropic's post says outright that it targeted memory corruption because it is easy to validate.
Multi-role and multi-tenant authorization. Testing authorization properly means holding a matrix of roles, tenants and objects, and asking for each cell whether the response is allowed. A 200 from another tenant's invoice endpoint is a critical finding in one product and intended behaviour in another, because that product has a shared-ledger feature. An agent can generate the requests. Deciding which responses are violations needs the same knowledge of intent that logic testing does.
Chains. Internal and Active Directory work is mostly about combining findings that are individually low or medium: a readable share with an old deployment script, a service account with a weak password, a delegation setting nobody reviewed, a path from there to a privileged group. Each step is ordinary, and the finding is the path that connects them. Agents enumerate these pieces well; the ARTEMIS result shows that. Deciding which of fifty mediums is worth three hours of follow-up, and when to stop, is judgment built from many engagements. Writing 53 Proving Grounds machines meant designing exactly these chains for other people to find, and the interesting ones are rarely the step a scanner flags first.
Safety on production. A real engagement runs against systems that make money. Knowing that a particular endpoint triggers an SMS to real customers, that a fuzzing run will fill a ticket queue, or that a password spray will lock out the finance team at month-end is part of the job. The Stanford team handled this with a human watching every agent session and a kill switch. Clients will need that same layer. I built Warden, a free MIT-licensed macOS guard for AI coding agents, because agents with shell access do things you did not ask for; an agent with network access to a client's production systems needs at least that much supervision.
Somebody has to sign the report
Regulated testing needs an accountable party. PCI DSS v4.0 requirements 11.4.2 and 11.4.3 require internal and external penetration tests by a qualified, organizationally independent resource, internal or external. Auditors working to ISO 27001 or to a banking regulator's rules ask a similar question in their own terms: who tested this, against what scope, and who stands behind the result if a breach later shows the test missed something. A model can draft that report. It cannot be the qualified party, and it cannot carry the liability. Even XBOW and Big Sleep put a human review step before anything was submitted.
What this does to the market
The part of the market that sells a scanner run with a cover page will get cheaper and mostly automated. Agents already do that work more thoroughly than the scanners did, and some of them validate their findings. Buyers who need a PDF for a checkbox will pay less for it, and they should.
The value moves in two directions. One is speed: AI-assisted delivery where recon, known-vulnerability checks and first-draft reporting take hours instead of days, so a client gets results within the week. The other is depth: human time spent on authorization models, business logic and attack paths, which is where the critical findings in a mature environment usually come from and where agents in the studies above were weakest.
Where I think the line sits
I already use AI when I write n-day PoCs. Given an advisory and a patch diff, a model reads the changed code quickly, proposes a trigger, and iterates on errors, which shortens the path from advisory to working exploit. That matches the UIUC result: the hard half is knowing where the bug is, and an advisory hands you that.
For a full engagement I think the work should split the same way. Recon, enumeration, checks for known CVEs, n-day reproduction and the first draft of the report are the parts AI shortens. The time saved should go to what the agents above missed: reading the application's roles and tenancy model, testing the business rules, chaining findings on internal networks, and deciding what is safe to run on a live system. The report still has to go out under a human name, because a client and an auditor need one.
None of this is fixed. Computer-use agents may close the GUI gap the Stanford authors describe, and the benchmarks will move. For now, AI has taken the parts of a pentest that a program can verify, and left the parts that need someone to understand what the business meant.
More of my notes are at yuznumara.com, and my lab machines are at labs.dakcyber.com.
Sources
- XBOW, Top 1: how XBOW did it, 24 June 2025
- TechRepublic, Google's Big Sleep Foils Hackers by Spotting SQLite Flaw Before Exploit, July 2025
- TechCrunch, Google says its AI-based bug hunter found 20 security vulnerabilities, 4 August 2025
- Anthropic Frontier Red Team, Zero-days, 5 February 2026
- Help Net Security, Anthropic: Claude Mythos identified 10,000+ software flaws, 26 May 2026
- DARPA, AIxCC results, 8 August 2025
- Fang et al., LLM Agents can Autonomously Exploit One-day Vulnerabilities, 2024
- Lin et al., Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing, 2025/2026
- Secure Ideas, What is required for a PCI penetration test?