Case Study: Verifying Security Fixes Instead of Trusting Them

The situation

A client maintained an open-source Nginx module that implements passwordless authentication using OpenPGP signatures. Visitors sign a server-issued challenge with their private key instead of using a password. The module handles session issuance, replay protection, per-IP rate limiting, and revocation checking, all in C, running inside the Nginx worker process.

The maintainer had already run two internal security self-audits and commissioned an external code-level review. By the time CyBax was engaged, the module had been through several rounds of fixes. The ask was not to find new bugs. The ask was to prove the existing fixes actually worked. That distinction shaped the whole engagement.

Why verification is different from review

Most security reports stop at the diff. A finding gets described, a patch gets proposed or applied, and the report says "fixed" based on reading the code. That step is necessary, not sufficient. A patch can look correct on the page and still fail at runtime. A config directive might not propagate the way the code assumes. A test suite might not exercise the edge case that matters. A build step might quietly do something different from what its own comments claim.

The CyBax approach on this engagement treated every "fixed" claim as something to test, not something to record.

Every finding was checked against the live repository state, not a summary of it. Every fix was built from source on the exact base images and dependency versions the fix specified, instead of assuming those versions were correct. The compiled binary was run against real traffic patterns, including the specific attack each finding described. Where possible, the underlying fault condition was reproduced directly, not just checked for the presence of a code path meant to handle it. Static and dynamic analysis ran on top of manual review. gcc -fanalyzer, cppcheck, AddressSanitizer, UndefinedBehaviorSanitizer, and the project's own fuzz harnesses all ran fresh against each candidate fix.

Several findings in this engagement were the kind of subtle, once-per-thousand-requests logic errors a human misses reading a diff and a sanitizer catches immediately.

Representative findings

The engagement covered two categories. One was application logic in the authentication module itself. The other was security of the build and release pipeline that packaged it. Below are samples from each, described generically to avoid identifying the project.

Application logic

Cross-boundary session reuse. The module supported multiple independent trust domains on one server. Think of a partner-facing area and an admin area, each trusting a different set of signing keys. Sessions were validated on every request by a cryptographic MAC, an expiry check, and a revocation check. The keyring a session was issued under was checked only at login and never again after that. Two locations could share a session secret, a documented and intentional configuration, while trusting different keyrings. They would accept each other's sessions without anyone noticing. A user who could authenticate in the low-trust area could present that same session cookie in the high-trust area and get admitted without ever appearing in its keyring. This was verified by minting a session in one trust domain and presenting it in the other. The session was rejected after the fix. Before the fix, the escalation worked.

Request smuggling through an undischarged body. A response generated early in the request lifecycle, before authentication was even evaluated, never read the incoming request body first. Nginx's own contract for that situation calls for the body to be explicitly discarded. The module didn't do that. The unread bytes sat in the connection buffer and got parsed as a second, pipelined request on the same keepalive connection. This was verified with a raw TCP socket sending one request whose body was itself a well-formed HTTP request, then counting response lines. Two responses came back on the unfixed build. One came back after the fix. This is a textbook request-smuggling primitive when the module sits behind a reverse proxy or CDN, which is exactly the deployment model it targets.

Weak identity extraction on a fallback path. The module pulled the signer's cryptographic fingerprint from the verification tool's structured output. The primary extraction path checked that the value was actually a hex fingerprint of the right length. A secondary fallback path, kept for compatibility with older verification-tool versions, checked only the length and not the content. A long enough non-hex value would get accepted as an identity and written into a session cookie, a structured log line, and a revocation comparison. Unvalidated subprocess output reaching a response header is a header-injection primitive. This wasn't exploitable against a standard, correctly behaving verification binary, but it was a real gap between two paths that should have enforced the same rule. This was verified with a stand-in verification binary built to emit exactly this malformed output. The exploit worked before the fix and failed after it, in an isolated test run.

Anti-automation throttle gaps, found across two review cycles. The module banned an IP address after a threshold of failed login attempts. The first finding involved two of the cheapest failure paths to automate. One was an unreadable request body. The other failed a structural pre-check before verification even ran. Both bypassed the counter entirely, so a client could hammer the login endpoint indefinitely and never trigger the ban. The second finding turned up a review cycle later, once that fix was itself scrutinized. The corrected logic had overshot in the other direction. It now counted the module's own internal faults, a failed subprocess fork or an exhausted memory pool, as if they were failed login attempts. A transient infrastructure hiccup could lock out every legitimate user for the full ban duration. This was verified live by forcing a genuine internal fault, an environment misconfiguration that reliably breaks a subprocess call, and confirming the client was never throttled across repeated attempts past the ban threshold. A real bad login attempt, run as a control on an identical instance, was banned exactly as configured.

Build and release pipeline

Signing key exposed to full network access. The container responsible for signing the project's package repository had the private signing key mounted for its entire runtime. That same container ran package-manager commands requiring full internet access for the whole time the plaintext key was present. This is the same failure shape behind at least one real-world supply-chain compromise of a widely used CI tool. This was verified by building the actual signing container and testing network reachability directly. Before the fix, DNS resolution and outbound connections succeeded the whole time. After the fix split the process into a network-enabled prep stage with no key present and a network-isolated signing stage, both failed outright during signing. Every tool the signing step needed still worked with no network at all.

Unpinned upstream dependencies. Base container images were referenced by mutable tag rather than an immutable content digest. Anyone with push access to the upstream image, or a compromised upstream account, could silently change what the next build pulled in, with no corresponding change visible in the project's own files. This was fixed by pinning to specific digests. It was verified by cross-checking the pinned values against what those tags currently resolve to on the public registry, and by building successfully from the pinned images directly.

Unverified third-party source download. The build process fetched a compressed source archive from an upstream project's official site over TLS, with no integrity check against that project's own published signatures. TLS protected the transfer but not against a compromised origin server or a tampered release artifact. This is the same mechanism behind a well-known backdoor incident in widely used compression tooling. It was fixed with a checksum gate against independently verified hash values. It was verified by downloading the real archive fresh and confirming the hash matched, then deliberately corrupting a copy and confirming the build refused it.

How each fix was actually proven

A few patterns from the methodology are worth calling out because they generalize to any codebase.

A fix might claim to exclude something, like an error type from a counter or a code path from a network. The strongest test is proving that exclusion negatively. Trigger the excluded condition repeatedly and confirm the expected behavior never fires. Run a working control case in parallel to confirm the mechanism being tested wasn't just broken outright.

When a fix touches build or release tooling, "does it still build" isn't the bar. That means rebuilding from the exact pinned dependencies over a real network. It means hashing real downloaded artifacts against the committed values too.

Environment quirks get worked around at the environment level, never by touching the code under test. In this engagement, that meant working around a network-inspection proxy in the test environment that interfered with fresh containers' TLS trust. Fixing that in the test harness kept the actual code under test unchanged.

A green test suite is a necessary condition, not a sufficient one. Several of the findings above existed in a codebase with a substantial passing test suite. The suite simply didn't cover the specific failure mode. New, targeted tests closed that gap. They weren't written just to confirm the existing ones still passed.

Outcome

Across the engagement, multiple authentication-bypass and session-integrity findings, all rated Critical or High, were closed and independently reproduced before and after the fix. A request-smuggling primitive was closed and reproduced with raw-socket testing. Supply-chain and build-pipeline hardening was applied and verified against live infrastructure. A pair of related throttling logic errors was caught a full review cycle apart, each verified with targeted fault injection rather than code inspection alone. Every fix was confirmed by execution. Compiled, run, and attacked. Not by reading the commit message.

This is a good fit for teams shipping security-relevant infrastructure code (auth modules, API gateways, anything parsing untrusted input in a systems language) who want confirmation that a fix actually holds, not just a second pair of eyes on the diff. If that's useful, CyBax Solutions offers this as a standalone verification/retest service on top of or independent from a full pentest — reach out at cybax.io.