We Audited Our Own Security Tool Before Asking Anyone Else to Trust It

Found two P0 bugs in our own code last week. Here's what that process looked like, what's still not done, and why I think building AppSec tooling requires a specific kind of honesty most launches skip.

The uncomfortable test

There's a specific kind of hypocrisy risk in building a security product: you're asking people to trust your tool to find flaws in their code, while your own code is sitting there, unreviewed, on the same trust. I didn't want to find out the hard way that we'd built exactly the kind of product we're warning people about, so before shipping anything further, I ran a real security review against our own codebase.

It found two P0-severity bugs. Neither is exotic. Both are exactly the categories of failure that show up over and over in research on how AI-assisted development introduces vulnerabilities — the same categories, in fact, that I'd been writing about in this build-in-public series.

What we found

The first was an SSRF gap: a scan-target URL, provided by the user, wasn't validated before the backend fetched it. Point that at a normal website, nothing interesting happens. Point it at a cloud metadata endpoint or an internal service instead, and you have a path to data or credentials that should never have been reachable from that request. This is the textbook SSRF pattern — the same one shows up constantly in published research on AI-generated code, because "fetch the URL the user gave you" is exactly the shape of function these models default to writing without validation unless someone specifically asks for it.

The second was broken access control: RBAC that looked implemented in the code but wasn't actually being enforced on some of the mutating routes. The difference between "we wrote authorization logic" and "we wired that logic into every code path that needs it" turned out to be a real gap, not a theoretical one. It's the same failure mode I'd written about weeks earlier when a real, disclosed CVE (a Row-Level Security gap on another AI-native dev platform) exposed hundreds of live apps: the model, or in this case the humans reviewing the model's output, optimized for "does the endpoint work" and didn't fully close the loop on "who's allowed to call it."

Finding both of these in our own product, days after writing publicly about exactly these categories of failure, was not a fun coincidence. It was also, I think, the correct outcome of actually looking.

What got fixed, and how fast

Both bugs were fixed and merged the same day they were found — SSRF hardening and RBAC enforcement, both merged July 2nd. Around that same work, several other issues that had been sitting quietly got cleaned up too: the logger was writing secrets and PII to plaintext logs, which now gets redacted before anything is written. Our rate-limiter, which was supposed to be protecting endpoints from abuse, turned out to be a silent no-op — implemented in name, doing nothing in practice, until we actually tested it under load and noticed. And the way requests were scoped to a specific workspace got reworked entirely, closing a path where one tenant's data could theoretically leak into another tenant's request context under specific conditions.

None of these are things I'm proud were there in the first place. All of them are things I'm glad we found before anyone else was depending on the product.

What's still not there, and why I'm saying so

The dashboard is filling in well — findings, fixes, reports, schedules, and notification pages all shipped in the same window as the security fixes. It's starting to look like a real product from the outside.

But the actual scan engine — the part of the product that does the thing the product is named for — is not deployed. The pipeline code exists now: target intake, the scan itself, findings persistence, a fix-PR generation step, and a retest step, all merged to main as of the last couple of days. That's real progress, and it's tempting to describe it as "done" in a build-in-public post, because merged code feels like a milestone.

It isn't the milestone that matters yet. Merged code has never run end-to-end against a real target. There's no verified scan result to point to. Until there is, saying the engine "works" would be exactly the kind of overclaim I'd criticize if a competitor made it, and exactly the kind of gap between marketing and reality that the underlying research I keep citing says is already the norm for AI-assisted development. I'd rather be a week later saying something true than a week earlier saying something that isn't.

Why this is the right order of operations

The plan from here is straightforward, if unglamorous: wire the pipeline together for real, and run the first genuine scan against our own codebase — the same one that just failed its own security review twice — before running it against anyone else's. If it can't hold up against something I built and understand completely, it has no business anywhere near a codebase I don't.

I think there's a broader point here for anyone building developer or security tooling specifically: the credibility of what you're selling is inseparable from whether you've actually subjected your own product to it. A project-management tool that's badly organized internally is merely ironic. A security scanner that hasn't found and fixed its own P0 vulnerabilities before asking anyone to trust it isn't ironic, it's a preview of what happens when a customer actually depends on it.

We found ours. We fixed them. The scan engine isn't live yet, and I'll say so again next week if it still isn't. That's the whole update.