Semgrep and SonarQube found zero of 232 access-control bugs

Both scored 0% on the vulnerability class attackers chain most. If a rule-based scanner is your main control, your backlog records what it couldn't see. Here's how to tell.

In the RealVuln v3.1 benchmark, two of the most widely deployed static scanners in the industry, Semgrep and SonarQube, found zero of 232 labelled broken-access-control vulnerabilities. Not a few. None. On missing authentication and authorisation, same result: zero of 173.

Broken access control is the number-one entry on the OWASP Top 10, and the class an autonomous attacker chains first. If a rule-based scanner is the main control standing between your code and that class of flaw, it isn't catching a reduced number of them. It's catching none.

That has a consequence most monthly security reviews get backwards. Every month a team presents a slide with one number on it: open vulnerabilities. If the number went down, the meeting goes well. But a backlog can only hold what a scanner detected, so the number on that slide answers a different question from the one the room thinks it is.

Disclosure: the figures below come from RealVuln v3.1, a benchmark built by Kolega Labs, where I work, and the metrics are adapted from a Kolega framework document. Both are public; the argument stands on the arithmetic, not on any product.

A thought experiment with real numbers

Three teams own identical codebases. Each runs a different scanner, then triages and fixes everything it reports. Which team has the most secure application?

The RealVuln corpus lets us put numbers on this. Its 140 repositories contain 4,138 labelled vulnerabilities. On that same code:

• Semgrep would have put about 410 into the backlog and missed 3,728, a 9.9% catch rate.
• The best general-purpose LLM (GPT Daybreak Blue) would have put about 3,383 in, missing 755.
• The security-specialised scanner (Kolega DevSec Max) would have put about 3,733 in, missing 405.

SonarQube, the other rule-based tool in the benchmark, also scored 0% on broken access control. The title's "zero of 232" is Semgrep's and SonarQube's shared result on that family; the detailed per-tool figures are in the report's Table 3.

The first team has the smallest backlog by a factor of eight, and the most unfixed vulnerabilities by a factor of nine. The backlog number says nothing about that.

Two caveats. The corpus is dense with vulnerabilities by design, so the absolute numbers don't transfer to your estate. And these scanners aren't your scanners. What does transfer is the shape: a queue can only hold what was detected, and on the same code, detection varies by an order of magnitude between tools.

The metric that moves the wrong way

Total findings open has a perverse property. It rises when detection improves and falls when detection degrades. Switch to a better scanner and your headline risk number goes up. Let a scanner silently fail on a large repository, as the last post showed can happen, and it goes down.

That makes it a reasonable workload indicator: it tells you how much triage and fixing sits in front of the team. As an exposure indicator it's poor. Keep it on the slide, but label it honestly.

What the backlog cannot tell an examiner

Remediation runs on a queue, and a queue is a register of the subset your tooling could see. That distinction becomes material when a regulator, auditor or insurer asks a simple question: how do you know the register is complete?

If the real answer is "we don't, because our tool is blind to the families an attacker would chain", then the backlog isn't a statement of your vulnerability position. It's a statement about your instrument's coverage. The next post looks at what the main financial regulators actually ask on this point, clause by clause.

Six metrics that survive an audit

Metrics fail audits for two reasons: they measure activity rather than exposure, or nobody can reproduce them. Each of these is defined closely enough that a third party could re-derive it from your records.

1. Detection completeness by family. The recall of your deployed scanning stack against a labelled corpus, per vulnerability family. Use a public benchmark, or the seeded-branch protocol from the last post on your own code. This is the metric that tells you what the backlog can and cannot contain.

2. Verified closure rate. Chains or findings closed by a failing re-test, divided by those confirmed in the period. A merged fix is not a closure. A later post in the series covers this.

3. Median time to verified close, by consequence band. Days from confirmation to a failing re-test. This answers the question an examiner actually asks: how long was it exploitable? CISA's BOD 22–01 is a useful reference point: it sets deadline-based remediation for known-exploited vulnerabilities in US federal systems, measured in days.

4. Residual reachable critical outcomes. The number of distinct protected outcomes (from the routes method earlier in this series) that known routes still reach at the control-plane or crown-jewel rings, and how many routes reach each. This states exposure in consequence terms, not severity counts.

5. Closure-set coverage. The share of in-scope protected outcomes for which you have computed the smallest set of fixes that interrupts every known route. It shows that remediation is planned against routes, not tickets.

6. Chokepoint concentration. The fraction of routes to a given ring that pass through a single finding or position. High concentration means one fix removes many paths, and one failure opens many.

Download the six-metric definition card (one-page PDF)

Presenting this without it sounding like an excuse

The risk in this argument is that it sounds like a team explaining why its numbers look bad. Three habits avoid that.

1. Lead with coverage, then workload. "Our stack detects an estimated X% of access-control flaws, measured on our own code. Within what we can see, the backlog is Y and falling."
2. Show the change in instrument as an event. When you upgrade detection, annotate the chart: "new scanner, backlog rose because we can now see more." Rising numbers with a cause are a sign of maturity.
3. Pair every count with a time. Not "12 criticals open" but "12 criticals open, median 9 days since confirmation, 3 past the 14-day target".

NIST's guidance on patch management and on security measurement makes the same underlying point: measure whether the risk response worked, not only whether it happened.

Next: what DORA, the FFIEC handbook and MAS's guidelines actually say about this, quoted rather than paraphrased.

References

1. Kolega Labs, RealVuln Benchmark Comparative Report v3.1, Table 2 and Section 6.
2. Kolega Labs, Continuous Architectural Resilience Framework v1.0, Section 6 and Table 4, 23 October 2026.
3. NIST, SP 800–40 Rev. 4, Guide to Enterprise Patch Management Planning, 2022.
4. NIST, SP 800–55, Measurement Guide for Information Security.
5. CISA, Binding Operational Directive 22–01: Reducing the Significant Risk of Known Exploited Vulnerabilities.