Your code scanners’ blind spot — measured

By Satyajith Mundakkal, with Agalya T, Mythily T M, Gowtham C P, and Santhosh Viswanathan | Companion to "Mythos, Measured"

We ran nine detector configurations across four detection paradigms — traditional scanners / static-dataflow analysers, general-purpose / code-capable models, a security-specialised model, and a tool-augmented review workflow — over three Python/FastAPI–React codebases. Their retained candidate findings overlapped surprisingly little. That supports layered discovery; it does not, on its own, establish which approach is best.

Status — exploratory

This is an exploratory comparative study of detector outputs, not a benchmark of detector accuracy. Unequal output sampling and the absence of blind adjudication prevent precision, recall, and detector-ranking conclusions; a separate 262-finding product-owner review does supply developer-validated false-positive rates, reported as scoped, non-blind results. Results are described as retained consolidated candidate records: one row in the canonical 334-row master after de-duplication/sampling, before validation, scoped to the tested configurations. The 334 rows encode 349 detector contributions because 14 rows carry multi-detector support. Data freeze: 18 Jun 2026.

Central thesis

The experiment does not establish which detector is best; the current data cannot answer that. Its defensible contribution is narrower. Under the tested configurations, different detection systems produced substantially different retained findings. Coverage is a property of the whole detection system — its analysis method, context, tooling, vulnerability intelligence, prompting, and verification — rather than of the model name. Low overlap between candidate outputs makes layering rational. The layer that matters most operationally is the one that converts a candidate into a validated, fixed, regression-tested, deployed risk reduction.

What surprised us. We expected the detectors to disagree. We did not expect the strict overlap to be this low: after consolidation, only 14 of 334 master records had more than one detector behind them. That shifted the recommendation. What began as a "which approach is best" question became a workflow-design one, and the honest answer stopped being "replace your scanner."

Why now

Security detection is changing with repository-scale Mythos-class models. For years, vulnerability detection meant a fixed toolchain: SAST, dependency and secrets scanners, dataflow analysis, run on a set cadence. Capable code models changed the ground under that: they can read a whole repository and reason about it in ways a rule-based scanner cannot. That raises the question every security team now faces: has detection fundamentally changed since Mythos, or has it simply added a new lane? And what does LLM-based detection actually add to the stack you already run? This study set out to map that gap on real production code: the space between what the traditional toolchain catches and what LLM-based detection surfaces. It is a gap-mapping exercise, not a contest. We compared what each approach found, and they mostly found different things. The gap is visible in this run; the response is to layer the old and the new, not to replace one with the other.

Why this study

What to add to an existing AppSec program

The companion brief, Mythos, Measured, argues that AI is collapsing the time and skill required to discover vulnerabilities while the machinery to fix them stays on human pace. That is a strategic hypothesis from the brief, not something three repositories can establish; we treat it as motivation, not as a result of this experiment.

The practical question for a CXO is narrower and answerable in part: "I have traditional application security testing today. What kinds of findings does each newer approach add, and how much do they overlap with what I already run?" This study addresses that in terms of retained candidate output and overlap, not accuracy.

Method

Four classes of detector, on three codebases

Three codebases vary size while holding the stack roughly constant. Nine detector configurations fall into four paradigms, colour-coded throughout. Treat the paradigm labels as an organising convenience: the experiment does not isolate whether a result is due to the base model, the harness, the tools, database access, context selection, the prompt, or a verification loop. The four-way split is a useful lens for describing what each kind of detector surfaced, not a claim that the label itself caused the difference.

Layer 1 — automated, before the master. Before any finding reached the candidate master, every detector ran a documented false-positive-reduction step: multi-pass scanning with automatic de-duplication and a mandatory verbatim-code-quote-plus-attack-scenario rule for the general-purpose LLMs (raw output roughly halved; Codex and Grok reductions ~43–53%), cross-tool confirmation and root-cause de-duplication in the traditional lane, and an at-source grounding rule for Fable. This removed the bulk of raw noise, but what survived are candidates, not confirmed findings. The per-detector mechanisms, the raw-to-deduped reduction chart, and the pre-validation estimate table are collected in Appendix C.

Layer 2 — human developer validation. A product owner then reviewed a 262-finding validation list and marked each finding Valid or False Positive. These are the developer-validated rates charted below. The result is the point of the two-layer design: automated reduction was necessary but not sufficient. After Layer 1, 29% of the reviewed findings (77 of 262) were still false positives, rising to roughly half for the general-purpose models (Codex 47%, Qwen 49%) and 64% for validated SonarQube, while the most tightly grounded lanes held (Trivy 0%, Fable 9%). Automated filtering narrows the pile; only human validation establishes what is real, which is why the workflow in Section 10 pairs layered discovery with an independent verification step before remediation.

False-positive rates · pre-validation estimate vs developer-validated

This is Layer 2 — the human pass. Each rate began as a researcher estimate from raw scan output (Layer 1); a product-owner review then validated 262 of the findings — a separate pass from the 334-record discovery master, not the same rows — letting each estimate be checked against a developer verdict for the first time. It is product-owner review, not the blind two-reviewer adjudication Appendix B specifies, and it measures false positives only: recall is still unmeasured.

Definitions used throughout

Consolidated master record — one row of the 334-row canonical master. The 334 rows encode 349 detector contributions (14 rows carry more than one detector: 13 two-detector, 1 three-detector), so "334 records" is not 334 separate emitted findings; the detector-output sheets hold a larger pre-master row count and are not the denominator for the main figures. Candidate — a finding before validation. Retained record — a candidate kept after the workbook's de-duplication/sampling. Strict match — two records linked only when they share location and consequence. Group — records sharing repository + CWE + raw category (a heuristic that can merge unrelated bugs). Paradigm — the four organising classes above; an attribute of the run, not a proven cause of coverage.

What constrains every number below

Unequal sampling. The three general-purpose models were each truncated to 50 retained rows; Fable (79) and Claude Code Security (57) were not. Cross-detector and cross-paradigm volumes are therefore not comparable, and no detector ranking is drawn. The caps can also affect observed overlap and category composition, so the low overlap is a property of the sampled retained set, not an estimate of detector-level true overlap. Partial adjudication. A separate 262-finding product-owner review now yields a validated false-positive rate (charted above); recall is still unmeasured, so coverage counts here remain contribution and overlap, not recall. Single run. One run per stochastic configuration; treat every figure as a single-run snapshot scoped to modern Python/FastAPI AI applications. Data version. The workbook also contains an older 368-record generation; only the 334-row master is used here. Availability. Claude Code Security is a research preview. Access to Fable / Mythos-class models was suspended on 12 Jun 2026 under a US export-control directive; the Department of Commerce lifted those controls on 30 Jun 2026, with Fable 5 returning globally from 1 Jul 2026 and Mythos 5 restored to approved US organisations. The Fable scan here predates the suspension, so its results stand; neither model is prescribed as an available production stage, and availability should be re-checked at publication time. A full run manifest (model slugs, prompts, commits, tool versions) is still required — see Appendix B.

The shape of the gap

An interpretive map of where findings landed

The map below organises the discussion: each category is placed by where, interpretively, it was caught.

The data behind the placement — which paradigm surfaced which category, by retained count:

Observed gaps in this configuration

Where each approach surfaced little or nothing

First, an honest look at the retained-record counts, and at the sampling that shapes them.

Gap one: the configured baseline surfaced few logic flaws

Across the three codebases, the configured traditional runs surfaced few authorization / IDOR and business-logic records, while the model-based reviews surfaced many such candidate records. Because the model candidates are unadjudicated and the samples unequal, this is a difference in retained output, not a measured recall gap. Independent literature (Section 9) reports low real-world recall for some SAST tool sets, but those results are language- and benchmark-specific and are not a baseline for this study.

Gap two: CVE provenance

The database-backed run (Trivy) accounted for the dependency-CVE records in the master inventory (13 IDs); one model run (Fable 5) also emitted two CVE IDs. A separate CVE matrix lists 19 IDs versus 15 in the master (an unreconciled discrepancy), and at least one ID is joined to the wrong product.

Withheld · CVE-coverage figure

We have withheld this figure rather than publish a number we cannot yet stand behind: a deliberate choice, not a gap. It awaits a full trace of every ID to its source detector output, package or product, advisory source (NVD / OSV / GHSA), repository commit, and affected-version evidence, and resolution of the 15-vs-19 and Trivy/Fable attribution conflicts. See Appendix B.

How much they agree

The corroboration-granularity problem

How often did more than one detector surface the same thing? The data answers two different ways, and the choice matters. (Two units recur below: a record is one row of the 334-row master; a group merges records that share repository + CWE + category; 145 groups in all.)

Anatomy of three findings

What contextual reasoning surfaced here

Three candidate records the configured pattern scanners did not surface. Each is the kind of issue that is a property of behaviour across requests, state, or history rather than a single dangerous line. All three are candidate records — not independently adjudicated or reproduced in this study.

Contribution under the matching rule

What each paradigm surfaced without cross-paradigm support

Stripping out the records that crossed paradigm lines (the 6 cross-paradigm agreements) leaves what each paradigm surfaced without corroboration from another paradigm: 328 records in all, the 334-record master minus those 6, split 55 / 139 / 77 / 57 across the four paradigms. These are paradigm-exclusive counts, not a performance ranking — records no other paradigm surfaced, same-paradigm agreements included, not detector-level exclusives. They are sampling-affected: the general-purpose total is built from three models each capped at 50 rows. The value is in the kinds of records each paradigm surfaced, not in comparing the totals.

Withheld · AI-vs-traditional headline percentage

It would be tempting to headline that ~82% of records were surfaced by an AI configuration and no traditional tool. The figure reconciles to the 334-row table, but because the underlying sampling is unequal (which affects de-duplication and category composition, not only volume), it is withheld until the model outputs are normalised under one rule. See Appendix B

A note on the security-specialised model

In this run the security-specialised model concentrated on cryptography, secret-exposure, privacy, and AI-specific records, including a "never decline" system-prompt issue that the other sampled model outputs did not surface in this run. This is suggestive of a distinct contribution, but a single model cannot establish a paradigm-level effect; a second security-specialised model and blind adjudication are required before any class claim.

Composition by repository

Category mix varied; this design cannot isolate size

Withheld · severity distribution chart

Better no chart than a misleading one. The severity chart is withheld because a defensible one cannot yet be drawn: the available severity data came from the older 368-record generation (its values summed to 366, not 334) and aggregated detector labels from different rubrics (for example SonarQube's Blocker / Major / Minor / Hotspot alongside Critical / High / Medium). Raw severity labels are not comparable across tools; severity will only be charted after a single normalised, adjudicated rubric is applied. See Appendix B.

More decision-useful than severity counts would be operational metrics this study did not capture: scan runtime and cost, analyst triage minutes per candidate, valid findings per 100 candidates, and time to verified fix. Those are listed as required follow-on work.

External evidence & its limits

What independent work does and does not support

Four external strands get cited alongside studies like this. Each helps only when stated precisely, and none is a baseline for this Python/FastAPI run.

• SAST recall (peer-reviewed). Li et al. (ESEC/FSE 2023, DOI 10.1145/3611643.3616262) found seven Java SAST tools detected ~12.7% of real-world vulnerabilities, most missed even when combined — language- and benchmark-specific context, not a baseline here.
• Vendor false-positive figure. Ghost Security (2025) reported ~91% false positives scanning Go/Python/PHP for three vulnerability classes — vendor-authored and narrow; it does not license "traditional tools miss the logic layer."
• AI discovery needs validation. Anthropic reported 500+ open-source vulnerabilities under triage; separately, the curl maintainer found five Mythos-labelled "confirmed" findings collapsed to one real bug after review — an argument for human validation, not a universal AI false-positive rate.
• Remediation telemetry. GitHub states Copilot Autofix clears two-thirds of supported alerts, and (separately) that median fix time fell from 1.5 hours to 28 minutes — distinct populations, not one causal claim.

They do not converge on a slogan; the only safe cross-cutting reading is that finding is necessary but not sufficient; validation and remediation carry the operational weight.

A code-centric discovery workflow

Layered by capability, not by product

Because the approaches mostly surfaced different records, completeness argues for layering. The stages below are capabilities, not a product ranking, and this is a code-centric discovery-and-verification workflow, not a complete application-security programme. It omits threat modelling, DAST / API testing, fuzzing, IAST, runtime / cloud posture, penetration testing, and human review.

Where a security-specialised model is used (stage 3), it should be read as "a security-specialised model, where authorized and available" — Fable / Mythos-class access was suspended on 12 Jun 2026 and, per Anthropic, restored from 30 Jun–1 Jul 2026 (Fable 5 globally; Mythos 5 to approved US organisations) — re-check availability at publication time; the Fable scan here predates the suspension. If prioritising, strengthen the verification capability (stage 4): converting candidates into validated, fixed, regression-tested risk reduction is the decisive step. This study does not provide the validated-risk, cost, and false-positive data needed to rank specific products, so no single-product recommendation is made.

The trade behind each layer

Qualitative and directional only — not benchmarked here, and not a ranking.

The hand-off

Discovery is the part this study measured

A note on what is, and is not, established here. Measured in this experiment: retained candidate records and their overlap. Imported from the companion brief: the finding-to-fix gap and the industry-scale evidence below. External / directional: the timeline figure. Author inference: the workflow recommendation in Section 10.

This study measured discovery records, not validation time, patch capacity, or remediation throughput. It therefore illustrates the brief's finding-to-fix gap but does not test it: we did not measure whether the resulting candidates could be serviced at any particular cadence. The brief itself reports industry-scale signals: external evaluators confirming a frontier model completing a 32-step network attack, and 271 fixes driven into a single Firefox release while the great majority of candidate findings stayed unpatched.

Where this paper hands off

Add detection layers by capability, but first instrument mean time from validated finding to verified-in-production fix, and the validation throughput behind it. Adding AI discovery on top of an un-instrumented remediation pipeline widens the gap rather than closing it. The maturity model and 30 / 60 / 120-day plan live in the companion brief.

Objections worth answering

The pushback, head-on

Isn't this just model noise dressed up as findings?

Partly, which is why these are called candidate records and why adjudication is required before any precision claim. But the candidate set here is not raw model output: every detector ran a documented false-positive reduction step before a finding reached the master inventory: multi-pass deduplication and mandatory verbatim-code-evidence requirements for the LLMs, cross-tool validation and root-cause deduplication for the traditional lane. The Codex raw-to-deduped ratio was roughly 2:1 (e.g. 247→122 on App A); Grok was similar. What remains are post-filter candidates, not hallucinations. The open question was whether survivors of those internal filters are true positives. A 262-finding product-owner review gives a first, scoped answer: it held best for the most tightly grounded lanes (deterministic Trivy at 0% and the security-trained Fable run at 9%) but confirmed the objection for the general-purpose models, where roughly half of the validated candidates were false positives (Codex 47%, Qwen 49%). Noise is not unique to models either: validated SonarQube was 64% on these codebases. The response is layering plus verification; verification is what removes that half, not picking a "clean" tool.

Won't one frontier model eventually do all of this?

This run does not settle that. One model run did emit two CVE IDs, but the database-backed lane accounted for the CVE identifiers in the observed table. CVE / SBOM provenance is a distinct capability (database access, not just reasoning), and remains necessary regardless of model strength.

Is "hybrid" just a model with a wrapper?

The design cannot isolate the cause. What is testable: integrated repository navigation, execution, history, and verification may change outcomes, but this study cannot attribute the hybrid's distinct records to any one component, and does not claim a false-positive advantage.

Three codebases, one stack, one run — why believe it?

Treat it as exploratory and scoped: Python / FastAPI AI applications, single run, unequal sampling, incomplete adjudication. The one result that survives that scoping is the low observed overlap, and it points the same way as the broader case for layering.

Conclusion

Not one tool — a layer of tools, plus verification

The single result that survives every caveat in this study is the low observed overlap: across these three codebases, the four paradigms surfaced largely different candidate records, with only 14 of 334 carrying more than one detector. Read conservatively, that says no single detector, and no single paradigm, saw the whole picture on these three codebases. Coverage came from combining methods rather than from choosing one. On the discovery question, the defensible conclusion is not "pick the best scanner" but "run a layer of them," because each surfaced a different slice of the candidate set.

The equally important half is what the overlap result does not license. These are candidate records, not confirmed vulnerabilities, and where they were adjudicated the false-positive share was substantial: roughly half of the general-purpose model candidates, and 64% of validated SonarQube findings, did not hold up. More detectors therefore mean more to triage, not automatically more real bugs fixed. Layering broadens what you find; it does nothing on its own to confirm or remediate. The value only materialises when each candidate passes through an independent verification step before it reaches a fix.

So the operational takeaway is a conjunction, not a single move: layer for discovery, then verify before you remediate. A conglomerate of complementary detectors widens coverage; independent verification converts that wider candidate set into fixed, regression-tested risk reduction without drowning the team in noise. This study measured only the first half, the discovery overlap, and argues the second half is where the operational weight sits. It does not rank the detectors, and it does not claim that any single one of them is enough on its own.

Limitations

What this paper is willing to be checked on

These are the open items a reviewer could still probe, distinct from the recomputation and source checks already completed in Appendix D. Appendix D records what was verified and held; the list below records what remains unresolved or untested.

1. Unequal / capped sampling. The three general-purpose models were each truncated to 50 retained rows; Fable and the hybrid were not. Detector and paradigm volumes are not comparable, and all ranking language is withdrawn.
2. Mixed data generations. The workbook contains an older 368-record generation alongside the 334-row master. Only the 334 master is used here; any figure that cannot be reconciled to it has been withdrawn (CVE coverage, severity).
3. CVE table not reconciled. 15 IDs in the master vs 19 in the matrix, conflicting Trivy / Fable attribution, and at least one wrong-product join. The CVE figure is withdrawn pending reconciliation.
4. Adjudication is partial. A 262-finding product-owner review provides validated false-positive rates (a separate pass from the 334 master, not blind two-reviewer adjudication; recall unmeasured). Detector-ranking and precision / recall language stays withdrawn pending blind adjudication; the false-positive rates are reported as scoped, non-blind results.
5. Single run; single stack. Non-deterministic models, one run each, three Python/FastAPI codebases. No generalisation to other languages or stacks is claimed.
6. No run manifest yet. Exact model slugs, prompts, commits, tool versions, and commands are required for reproduction (Appendix B).
7. What would change the read. A validated ground-truth set showing high inter-paradigm overlap, or a traditional toolchain matching model logic-finding recall under adjudication, would undercut the layering thesis. Neither appeared here — but neither was decisively tested.

What's next

Can AI break in?

This study measured what each approach finds. The next asks what an attacker could do with it. The question is not whether a model can exploit a single weakness in isolation; scanners already flag those one at a time. It is whether a model can take the defects surfaced here and chain them — combining several across a codebase into a working path to access. That is the distinctive advantage of an AI attacker: a human tester usually works one finding at a time, while a model can reason over many at once and stitch even low-severity issues into a real breach. Our next paper puts that to the test, driving models to combine the identified vulnerabilities and confirm which chains are genuinely reachable, under authorized and controlled conditions. It is the sharpest form of the verification layer this paper argues for.

Appendix

A. Method & reproducibility

Enough to interrogate the current numbers. The underlying records live in the companion workbook (334-row master).

B. Unresolved data & required work

What must be completed before the withdrawn figures and claims can be restored, and before this becomes a benchmark rather than an exploratory snapshot.

Run manifest — required fields and current status. The fields a full reproduction needs, and what this snapshot already documents versus what is still outstanding.

Withdrawn pending the above: the former CVE-coverage figure, the former severity figure, all detector-ranking statements, the AI-vs-traditional contribution headline, and precision / recall claims. Developer-validated false-positive rates are now reported (Section 2) as scoped, non-blind results from the 262-finding pass, distinct from the 334-record discovery master.

C.False-positive reduction, by detector

The per-detector mechanisms behind the false-positive-reduction step summarised in Section 2, the raw-to-deduped reduction chart, and the pre-validation estimate table. These record the research team's read of the elimination step applied before findings reached the master inventory; the developer-validated rates charted in Section 2 are the measured payoff.

How each model reduced false positives before findings were counted

Qwen3-Coder-30B treated each candidate as a hypothesis: the workflow gathered supporting evidence from related files, functions, trust boundaries, and attack paths, re-evaluated the finding in that wider context, recomputed a confidence level, and classified it as Confirmed, Likely, Needs Verification, or False Positive. Cross-tool validation was also applied: a finding was treated as confirmed only when a second tool independently flagged the same file and line. Researcher-estimated FP rate: 10–20% of retained candidates (over-flagging on logging patterns and unconfirmed auth-logic chains).

GPT-5.3-Codex ran four sequential passes — per_file (each file independently), deep (cross-file call-chain tracing), manifest (config and dependency files), and holistic (full cross-cutting attack chains) — with automatic deduplication collapsing any root cause flagged by more than one pass. Every surviving finding was required to include a verbatim code quote and a numbered attack scenario before it was written to output; findings without traceable code evidence were dropped. Raw-to-deduped reduction: 247→122 (App A), 275→146 (App B), 327→188 (App C), approximately 43–51%. Researcher-estimated FP rate: 10–15%.

grok-4.3 ran two passes (per_file then deep) with the deeper-context version replacing the narrower per-file version where both flagged the same root cause. Same mandatory code-evidence requirement as Codex. Raw-to-deduped reduction: 107→53 (App A), 101→47 (App B), 129→62 (App C), approximately 50–53%. Researcher-estimated FP rate: 10–15%.

Fable 5 did not preserve a separate raw-output set in this study; its source reports were already filtered by a grounding rule that required a verbatim code quote pulled directly from the file and a step-by-step numbered attack scenario before a finding was written, so anything that could not be grounded in actual code was not recorded. The priority-first deep file read (fewer files, read in full) appears to have further reduced context-free flagging. That filtering appears to have reduced speculative output, but it did not eliminate false positives: the researcher prior was ~5%, the lowest estimate of any model-based detector, near the deterministic traditional scanners (GitLeaks 0–5%, Trivy ~5%); the later product-owner validation observed 9%. Treat grounding as a useful discipline, not as proof of correctness.

All rates above are researcher estimates from direct observation of scan output for these codebases, not independently adjudicated. They document that raw output was substantially filtered before any finding reached the 334-row master inventory, but they do not license a measured precision or recall claim.

D. Independent data-validation & source verification

This appendix records an independent pass over the paper against the raw data files used to build it and against every external figure it cites. Each computable number was recomputed from the primary workbook; each external claim was checked at its origin. Results are reported whether or not they are favourable

Internal recomputation — charted figures vs the 334-row master

Every quantitative figure derived from the 334-row master inventory was recomputed independently and matched: record structure (334 records: 320 single-detector, 13 two-detector, 1 three-detector; 349 detector contributions); per-detector retained counts (Fable 79, Claude Code Security 57, Qwen / Codex / Grok 50 each, Semgrep 22, SonarQube 16, Trivy 15, GitLeaks 10); paradigm exclusives (Traditional 55, General 139, Security 77, Hybrid 57); the category×paradigm matrix, repository×paradigm table, group counts; and the Codex / Grok raw-to-deduped reductions. No figure derived from the master diverged from it. The App A/B/C footprint table is not fully re-derived from the master (it draws on a separate comparative-analysis source), and the App C Kubernetes-manifest count remains a source conflict, flagged separately below.

Developer-validated FP-rate provenance

The validated false-positive rates in Section 2 trace to the updated 262-row validation list (185 valid / 77 false positive overall); all nine per-detector rates reproduce exactly from that file. An earlier adjudication of the same 262 findings (182 valid / 80 false positive) also exists and is superseded: it differs by exactly three findings (one each from Codex, Fable, and Qwen), later re-adjudicated from false positive to valid. The paper uses the later figures throughout. A reader cross-checking the earlier summary should therefore expect Codex 50%→47%, Qwen 52%→49%, and Fable 11%→9%; the discrepancy is a data-version difference, not a computation error.

Residual discrepancy carried forward

One source conflict remains unreconciled: the App C footprint line reports 16 Kubernetes manifests (from the comparative-analysis source), while the Fable primary review of the same repository states it contains no Kubernetes, Helm, or Terraform manifests and deploys through an Azure Web App pipeline despite a repository label suggesting Kubernetes-based deployment. The two source documents disagree and the comparative-analysis source could not be independently re-derived here, so the figure is left as reported and flagged for reconciliation with the run-manifest work in Appendix B.

This validation covers arithmetic reproducibility and source attribution only. It is not a re-run of the detectors and is not a full reproducibility package: it does not re-execute any scan, does not measure detector recall, does not re-adjudicate any finding, and does not convert the researcher-estimated rates in Appendix C into a measured precision. A complete run manifest (Appendix B) is still required to reproduce the scans themselves.

Sources & companion material

Primary data: the 334-row Vulnerability Research Workbook master inventory (data freeze 18 Jun 2026). Strategic framing: Mythos, Measured — A CXO Brief on AI-Powered Cybersecurity (2026).

External evidence (context only, not baselines for this study): Li et al., "Comparison and Evaluation on SAST Tools for Java," ESEC/FSE 2023, DOI 10.1145/3611643.3616262 (the 12.7% figure); Ghost Security 2025 (vendor false-positive figure); Anthropic, Claude Code Security; GitHub, Copilot Autofix telemetry (github.blog, 2024); D. Stenberg / curl, maintainer review of a Mythos-reported finding (daniel.haxx.se, May 2026). Methods differ; figures are not directly comparable to this study. External figures were checked against primary sources where available; where only vendor-authored or secondary summaries were used, that status is labelled and the figure is treated as contextual only; see Appendix D.

References

1. Kaixuan Li et al., "Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java." ESEC/FSE 2023. doi.org/10.1145/3611643.3616262. Accessed 5 Jul 2026.
2. Ghost Security, Exorcising the SAST Demons (2025), as summarised in Help Net Security, "91% noise: A look at what's wrong with traditional SAST tools," 19 Jun 2025. helpnetsecurity.com/2025/06/19/traditional-sast-tools. Vendor-authored; contextual only. Accessed 5 Jul 2026.
3. GitHub, "Found means fixed: Secure code more than three times faster with Copilot Autofix," github.blog, Aug 2024. github.blog — secure-code-more-than-three-times-faster-with-copilot-autofix. Accessed 5 Jul 2026.
4. Daniel Stenberg, "Mythos finds a curl vulnerability," daniel.haxx.se, 11 May 2026. daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-vulnerability. Accessed 5 Jul 2026.
5. Anthropic, "Redeploying Claude Fable 5," 30 Jun 2026 — states that export controls on Fable 5 and Mythos 5 were lifted, with Fable 5 returning globally from 1 Jul 2026 and Mythos 5 availability limited to approved US organisations. anthropic.com/news/redeploying-fable-5. Accessed 5 Jul 2026.
6. Anthropic, "Making frontier cybersecurity capabilities available to defenders" (Claude Code Security research-preview announcement), Feb 2026. anthropic.com/news/claude-code-security. Accessed 5 Jul 2026.

Security & handling note

Repository identities are anonymised throughout: described by neutral labels (App A / B / C) and footprint, never by name or function. No secrets, credentials, or personal data are reproduced anywhere in this paper; the tables and charts report only aggregate, de-identified results. Any sensitive findings surfaced during the study were disclosed privately to the owning teams and handled under responsible-disclosure and data-protection practices, outside this public write-up.

AI & Cybersecurity research · exploratory snapshot: retained candidate records, scoped to the tested configurations, pending blind adjudication and a full reproducibility manifest.