Can an AI Agent Replace Your Code Security Scanner? I Tested It.

Security teams are being asked a new question: "We already pay for a code scanner. Do we need an AI agent as well, or can the agent just replace the scanner?"

Vendors and demos give confident answers in both directions. I wanted an answer I could check myself, on code where I knew what the bugs were.

The two approaches work very differently, which is why the question is hard. A scanner applies rules and, in the better ones, follows data flow from user input to a dangerous call. It is fast, repeatable and easy to run in a pipeline, but it only finds what its rules describe, and it cannot tell you whether a finding is actually exploitable in context. An agent reads the code more like a human reviewer. It can follow intent, connect two small bugs into one serious one and explain the fix in plain language, but it is not deterministic, it cannot prove anything, and it can make changes you did not expect.

Scanners also disagree with each other more than you might expect. If you run only one, you may be getting half the picture without knowing it. So the real question became three questions: how much do the scanners overlap, what does an agent add, and what happens when the agent starts changing code?

For the test I used Demo_VulnCode by Filipi Pires, an intentionally vulnerable repo with five small apps in Node, Python, Java, Go and PHP. The bugs are planted on purpose and labelled in the source. I ran three scanners on every app: Snyk Code, which does data-flow analysis and has a free tier, Semgrep OSS, which is free and mostly pattern based, and Drogonsec, a young open source scanner at version 0.4.1. Then I asked Claude Code to review the same code as an agent and apply its own fixes on a branch, with one commit per vulnerability class, and I scanned everything again.

Here is what I wanted to find out:

Which tool catches which kinds of bugs, and where do they disagree?
Does an agent find problems that scanners cannot?
When an agent fixes the code, do the scanners agree it is fixed?
What goes wrong along the way?

The five app folders in the repo, or the command window where you started the scans.

The short version: Snyk found almost everything, the two free tools each found a different half of the problem, and the AI agent saw things no scanner can. The longer version is more useful.

How I scored it

Raw finding counts are misleading. One bug can trigger three overlapping rules on the same line, and one noisy dependency check can add ten findings by itself. Across the five apps, Snyk Code reported 73 findings, Semgrep 59 and Drogonsec 45, and those numbers say very little.

So I counted distinct vulnerability classes per app, such as SQL injection or path traversal, against the bugs planted in the source. That came to 39 classes. I left out two labelled bugs that are not actually exploitable: an XXE in the Node app and a "weak token" in the Go app that uses a secure random source.

Distinct planted vulnerability classes caught across the five apps. The Claude Code score is not a fair benchmark, as explained below.

What each tool did

Snyk Code caught 38 of 39. Its strength is data flow. It follows user input from a request parameter through helper functions to the dangerous call, which is why it caught the SQL injection, SSRF and path traversal bugs that pattern matching missed. Its only miss was a missing authorization check in the Go app, which requires knowing what the code is supposed to do. Its weakness is volume. It reports XSS generously, so some of its 19 findings on the Python app overlap.

Snyk Code summary for one app, for example the python-app test summary showing 19 open issues.

Semgrep OSS caught 26 of 39. It is fast, free and needs no account. It did well on Python (8 of 9 classes) and poorly on Node (4 of 7), where it missed SQL injection, SSRF and path traversal. It also reports one bug several times: its 24 findings on the Python app are about ten issues. The free ruleset is mostly pattern based, so it does not follow data across functions the way Snyk does.

Semgrep scan summary, for example 'Findings: 24' for the Python app.

Drogonsec caught 19 of 39. It is an open source scanner at version 0.4.1, with readable output that includes CWE, OWASP and CVSS data and a suggested fix. It is good at hardcoded secrets and configuration problems and weak at injection bugs. Its Node dependency scan reported 10 issues by reading package.json files inside node_modules, while Snyk and npm audit both reported none. It is a young project, so I treat it as a supplement today, not a replacement.

Drogonsec findings output with the CWE, OWASP and Fix lines visible.

Semgrep and Drogonsec together caught 34 of 39. On the PHP app they split the bugs almost perfectly in half and together covered all eight classes. If you are limited to free open source tools, run both.

Claude Code scored 39 of 39, with an asterisk. The source files carry comments that label each planted bug, so the agent effectively had the answer key. Ignore the score. What matters is how its review differed from a scanner:

It rated findings by real exploitability. In the Java app, Runtime.exec with a single string does not start a shell, so the real risk is argument injection, not command chaining.
It chained bugs. Flask debug mode plus a path traversal lets an attacker read the files the debugger PIN is derived from, which turns a file read into code execution.
It found a bug no scanner flagged. In Express 4 on Node 24, a rejected fetch inside an async handler is not caught, so one bad request to the proxy route can crash the server.
It called two labels false positives instead of repeating them.

A short excerpt of the Claude Code audit output showing a finding with file, line and severity.

Then I let the agent fix it

I created a branch and asked Claude Code to apply its own fixes: one commit per vulnerability class, no changes to unrelated code, and stop at anything that needs a design decision, like adding authentication. Then I re-ran all three scanners with the same settings.

Total findings across the five apps. Counts overstate progress, because some remaining findings are noise on code that is already fixed.

SQL injection is gone in all five apps. So are the MD5 hashing findings, the Flask debug mode and template injection, and the hardcoded secrets, apart from one low finding left in Java. Snyk went from 73 findings to 34 and Semgrep from 59 to 23.

Four things were still open. The unsafe deserialization points are all still there: eval in Node, pickle in Python, ObjectInputStream in Java and unserialize in PHP. Each one is flagged by at least two tools. Replacing them changes the input format, which is exactly the kind of design decision I told the agent to stop at.

The git log of the security-fixes branch (git log — oneline main..security-fixes) showing one commit per vulnerability class.

Three surprises

The fixes added a vulnerable dependency. The Go app gained golang.org/x/crypto v0.33.0, most likely for password hashing in place of MD5. Drogonsec reports 21 advisories against it. Most are in ssh packages the app probably never imports, but nobody chose that version on purpose. Without those 21 findings, Drogonsec's total fell from 45 to 26. An agent that fixes code can also add dependencies, so rescan dependencies after every AI-generated change.

Scanners flagged fixed code. Semgrep and Drogonsec still reported lines that already call html.EscapeString, htmlspecialchars and escapeshellarg. Snyk reported the same Go path traversal and SSRF findings as before, with the same finding IDs and only the line numbers changed. Either the scanner does not recognize the new validation helper or the flaw remains, and a scan cannot tell you which. Only reading the diff can.

Fixing code did not fix dependencies. The vulnerable Flask, requests and MySQL connector versions were untouched. The agent did what the code audit said and nothing more.

One smaller lesson came from my own process. Drogonsec flagged its own earlier result file, which I had saved inside the Java app folder, as containing leaked secrets. Keep scan output outside the folders you scan.

What I would tell an AppSec team

Use a data-flow scanner as the base, and add a free pattern scanner as a second opinion.
Use an agent for triage, exploitability and attack chains. Its value is judgment, not volume.
Never accept a drop in finding count as proof of a fix. Review the diff.
Rescan dependencies after every AI-generated change.
Decide ahead of time which fixes need a human, such as authentication, CSRF protection and changing a data format.

The limits of this test

This is an intentionally vulnerable repo with tiny apps. Real code is messier.
Each tool ran once with default settings on free tiers. Semgrep was not logged in, so no pro rules, and Drogonsec's optional AI remediation was off.
I scored the results by reading the output, and I counted classes, not every individual finding.
Claude Code could read the bug labels in the source, so its score is not comparable to the scanners.
Drogonsec is an early version of an open source project, and credit to Filipi Pires for building it and for the demo repo. Its results will likely improve.

This is not a vendor ranking. It is one engineer measuring what these tools do on the same code, and the main result is that the overlap between them is smaller than most teams assume. What would you want measured next?

What I would test next

This was a first pass, and it leaves obvious gaps. Here is what I would add to make the comparison fairer and more useful:

Run the agent blind. Strip the CWE comments from the source so the agent has no answer key, then score it against the scanners on equal terms.
Repeat the runs. Agents are not deterministic, so I would run each tool several times and measure how much the results change.
Use real code. Apply the same test to a messy open source project with published advisories, where the ground truth comes from real CVEs instead of planted labels.
Combine them. Feed the scanner findings to the agent for triage and check whether the false positives drop while the real bugs stay.
Demand proof. Ask the agent to write a proof of concept for each finding and run it in a sandbox, so the claim is verified instead of asserted.
Test dependencies properly. Re-run Snyk Open Source and compare reachability, not just version matches.
Review every fix. Read each diff, add tests, and confirm the agent did not break behavior.
Add more tools. SonarQube and other scanners belong in the same table.

Where I think this is going

These are predictions, not facts, and I could be wrong. They come from one small experiment and from the work I see in application security every day.

Scanners become the evidence layer and agents become the judgment layer. Scanners are repeatable and auditable, which is why they stay in CI. Agents are good at what scanners lack, which is context, exploitability and attack chains. In my test each side was strong exactly where the other was weak.
Triage is the first job agents will really take over. Finding bugs is not the bottleneck. Deciding which findings matter is. Semgrep and Drogonsec both flagged lines that already escaped their output, and reading a finding in context is work an agent does well.
"Fixed" will need proof. A finding that disappears from a scan is not a fix, and a finding that stays may be a scanner that cannot see a custom validator. I expect agent patches to come with a test or proof of concept that fails before the change and passes after it, plus a re-scan.
AI-authored changes will get their own guardrails. My agent added a Go dependency with 21 advisories while fixing a hashing bug. Expect policy checks on AI pull requests that flag new dependencies, secrets and permission changes, and require a human sign-off.
Benchmarks will get harder to trust. Labelled vulnerable repos come with answer keys, and public ones may end up in training data. I expect more evaluation on private, unlabelled code and on real advisories.
The agent itself becomes part of the attack surface. An agent with shell access and repository rights can be misled by a malicious code comment, given too many permissions or fed a poisoned dependency. Whoever scans the code will also have to scan the scanner.
Humans keep the decisions that change behavior. In my test the agent left the four deserialization sinks and the missing authentication alone, which are the places where someone has to choose a new design. I expect that to stay true for a long time.

So, can an AI agent replace your code scanner?

My answer today is: not on its own. In the next few years I expect agents to take over the reading and the paperwork around scanning before they replace the scanners themselves. What would change my mind is an agent that matches scanner coverage on unlabelled code, gives the same answer across repeated runs, and proves each finding with a working exploit.

Contents