I Built an AI Agent That Patches Our Infrastructure — Here’s What I Learned About Trusting an LLM…

Every security team knows this feeling: a vulnerability scan comes back with a few hundred findings, most of them routine, a handful genuinely dangerous, and no good way to tell which is which without someone manually reading every single one. Multiply that across a fleet of Linux servers and Windows machines, repeat every scan cycle, and you have a full-time job that nobody actually wants.

I spent the last few weeks building an agent to do this triage — and, more interestingly, to act on it. Not just flag vulnerabilities, but actually patch the safe ones automatically, on real infrastructure, with a human still in the loop for anything that isn't obviously safe. Here's how it works, what broke along the way, and what I'd tell anyone else attempting this.

The problem with "just patch everything"

The naive version of this project is trivial: parse the scan, run apt-get upgrade on every flagged package, done. Nobody does this, for good reason. Some patches are routine point releases with zero blast radius. Others — a kernel update, a database engine major version bump, patching the configuration management agent itself — can take down a production service if applied blind. A patch pipeline that can't tell these apart isn't automation, it's a liability generator.

So the actual problem isn't "how do I run apt-get remotely." It's "how do I decide, at scale, which of five hundred findings are safe to touch without a human reading each one" — and then act on that decision responsibly.

The architecture

The pipeline has four stages:

1. Fetch. The agent pulls scan results directly from the vulnerability scanner's own API using an access token — no manual export/import step. This matters more than it sounds: scan data is a moving target. A scan taken last week and remediated this week can already be stale if the vendor shipped a newer patch in between (more on this later — it bit me).

2. Categorize. Findings get bucketed by how they're actually fixed — Linux OS packages (apt/dpkg), Python packages (pip — genuinely different from apt, since a package manager has zero visibility into what pip installed), Windows OS patches, and specific browser/app upgrades. Each category needs a completely different remediation mechanism, so this split happens early.

3. Assess. This is the interesting part. Every group of affected packages/hosts gets sent to an LLM with a simple, deliberately narrow question: does this look safe to auto-apply? The prompt is explicit that the model has no live telemetry — no view of what's actually running on the machine, no dependency graph, nothing but general knowledge of the package and whatever context (host role, criticality) you give it. It's told, in no uncertain terms, to answer "needs review" rather than guess "safe" when it isn't confident.

This produced some genuinely good judgment calls. The model correctly flagged a security patch to the fleet's own configuration-management agent as risky — right call, since patching that tool means briefly losing your own ability to manage the fleet while it restarts. It flagged glibc updates as needing review, because glibc underlies everything and a bad interaction is a fleet-wide outage, not a single-host problem. It correctly treated routine library patches (an XML parser, an image codec, a text editor) as safe, since blast radius there is naturally contained.

4. Execute — with a human still choosing what runs. Nothing executes automatically. The pipeline generates a plan showing exactly what it would do, grouped by the model's own assessment (safe / risky / needs review), with a one- or two-sentence rationale for each. A human reviews it, picks specific findings to act on, and only those get executed — everything else is skipped, visibly, with the reason shown.

Building in the honesty, not just the automation

The thing I kept coming back to: an LLM confidently sounding right is not the same as an LLM being right. A few design choices came directly out of that discomfort:

• Fail closed on missing data. If the model can't be reached (no credential, API down, whatever), the pipeline doesn't default to "safe" — it defaults to "needs review." Silence should never look like a green light.
• No exact-version pinning. Early on, the tool pinned installs to the exact version the scan reported as "fixed." This broke in a genuinely instructive way: by the time remediation actually ran, the vendor's patch repository had already moved past that version to a newer one — so the pinned version simply didn't exist anymore, and the install silently failed. The fix was to stop treating the scan's reported version as a target and instead just take "whatever the current patched candidate is," trusting the repository rather than a stale snapshot.
• No compromise on supply-chain trust for browser upgrades. The first version used a popular third-party package manager to upgrade Chrome and Firefox. It worked, until a routine browser release broke the package manager's own pinned checksum for that installer — a known, recurring failure mode for community-maintained packages tracking vendor software that updates in place. Rebuilt it to download installers directly from the vendor's own domain and cryptographically verify the code-signing signature before ever running it — fail closed if the signature doesn't check out, no fallback, no "ignore checksum and proceed anyway."
• An audit trail that isn't an afterthought. Every login, every override of the model's judgment, every execution — who, what, when — logged from day one. If this system is going to make judgment calls about production infrastructure, the paper trail needs to be as strong as the automation.

What actually broke, and what that taught me

The most educational failures weren't in the AI logic at all — they were in the unglamorous plumbing of talking to old, well-worn infrastructure.

One configuration-management server in the fleet was old enough that its automation tool streamed results as multiple separate JSON messages instead of one combined document. My first parsing pass assumed a single document, silently failed on the mismatch, and reported zero results — even though the underlying data was sitting right there in the output. The fix wasn't more AI, it was writing a parser robust to malformed assumptions and, just as importantly, making failures loud instead of silently returning nothing.

Another lesson: even "the job succeeded" isn't always true. I later diffed a before/after scan across the whole fleet to verify the agent's own work, rather than trusting its self-reported success. Out of roughly 130 patch operations, two machines had non-cooperating package repositories (expired signing keys, broken TLS to a legacy mirror) that silently blocked the update step entirely — invisible in the tool's own output, only visible by independently checking whether the actual installed version had moved. That verification step is now a permanent part of how I trust this system: never just trust "exit code 0," always check the world actually changed the way you expected.

Does this extend to containers and Kubernetes?

Short answer: the pattern generalizes cleanly. The remediation model doesn't, and that distinction matters.

Container and cluster vulnerability scanners produce the same shape of structured findings a host scanner does — parseable, categorizable, assessable by the same kind of "does this look safe" question. But the actual fix looks completely different. You can't patch a package inside a running container and call it done; containers are meant to be immutable. The correct remediation is: rebuild the image with the updated package or base image, push it, and roll it out — via a deployment update or your existing CI/CD pipeline — not an in-place install. Cluster-level configuration findings (benchmark failures, RBAC misconfiguration) are a third category again, fixed by applying corrected manifests, touching no image at all.

So the categorization-and-judgment layer of this architecture — parse, group, ask "is this safe," gate on human selection, audit everything — transfers almost unchanged. The actuation layer needs a genuinely different tool per remediation model: one that rebuilds and redeploys images, one that applies cluster manifests. It's a real, buildable extension, not a stretch — but it's an additional tool to build, not a config flag to flip.

What I'd tell someone building this today

Treat the LLM's judgment as a first filter, never the last word. Make "I don't know" a valid, common, expected output — not a failure mode to engineer around. Verify outcomes independently rather than trusting self-reported success, especially against infrastructure old enough to have its own accumulated quirks. And expect that most of your debugging time won't be spent on the AI at all — it'll be spent on the same unglamorous plumbing that's always made infrastructure automation hard, just now with an LLM sitting at one stage of the pipeline instead of a static ruleset.

The automation is the easy part. Trusting it responsibly is the actual project.

If you're working on similar problems — automated remediation, AI-assisted infrastructure decisions, or just trying to get a handle on patch fatigue at scale — I'd love to compare notes.