I Reduced My Container CVEs by 90% — Then Everything Broke

How migrating to minimal images for security introduced a new class of runtime failures

Vedant Shrotria

~9 min read · February 13, 2026 (Updated: February 13, 2026) · Free: Yes

My security scanner was screaming. The golang:latest base image I was using had 847 known vulnerabilities. The fix seemed obvious: switch to a minimal image like scratch or distroless. Zero OS packages means zero CVEs, right?

I made the change, pushed to production, and watched my service crash-loop.

The culprit? A simple exec.Command("curl", ...) call that worked perfectly in the full image but failed silently in the minimal one — because curl doesn't exist there.

Whether you're migrating from ubi9 to ubi9-micro, from debian to distroless, or going all the way to scratch, the same class of issues can bite you.

This is the hidden cost of container security hardening.

When you migrate from full-featured base images to minimal ones to reduce your attack surface, you're also removing binaries, shells, and libraries that your code might silently depend on. Moving from ubi9 to ubi9-micro? You lose bash, curl, and most coreutils. Going to scratch? You lose everything.

These dependencies don't show up in your go.mod. They pass all your tests. They only fail in production.

After breaking my deployments one too many times while chasing a clean vulnerability report, I built go-runtime-compat — a static analyzer that catches these compatibility issues before you deploy.

The Security vs. Compatibility Tradeoff

The container security playbook is clear:

The math is compelling. Fewer packages = fewer vulnerabilities = smaller attack surface. Security teams love it. Compliance audits pass. Trivy reports go green. Red Hat's UBI-micro gets you close to scratch while still being RHEL-compatible.

But there's a catch.

When you remove packages to eliminate CVEs, you also remove functionality your code might depend on.

The Usual Suspects

1. External Binary Dependencies

// This works on your Mac, fails in scratch
cmd := exec.Command("curl", "-s", "https://api.example.com/health")
output, err := cmd.Output()

Your Go code compiles fine. Your tests pass. But curl doesn't exist in a scratch image. The error only surfaces when that code path executes in production.

2. CGO and libc Mismatches

import "C"  // This innocent line can break everything

CGO binaries are dynamically linked against glibc. Alpine uses musl. Scratch has nothing. The result? Segfaults, missing symbol errors, or binaries that simply won't start.

3. Shell Commands in Shell-less Containers

exec.Command("bash", "-c", "echo $HOME && do-something")

distroless images don't have a shell. Neither does scratch or ubi9-micro. That "simple" shell command becomes an instant crash.

4. The Dockerfile Disconnect

My Go code and Dockerfile lived in the same repo but were developed separately. The code assumed certain binaries exist. The Dockerfile assumed the code was self-contained. Neither validated the other — and security scanners certainly don't check for this.

Building go-runtime-compat

I needed a tool that could answer one question: "If I switch to this minimal base image, will my code still work?"

So I built go-runtime-compat — a static analyzer that bridges the gap between security hardening and runtime compatibility. It analyzes your Go source code and your Dockerfile together, cross-referencing code dependencies with container capabilities to catch mismatches before deployment.

How It Works

The tool operates in three phases:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Go Analyzer   │     │   Dockerfile    │     │   Correlation   │
│                 │     │    Analyzer     │     │     Engine      │
│ • exec.Command  │────▶│ • Base images   │────▶│ • Cross-ref     │
│ • CGO imports   │     │ • Build stages  │     │ • Compatibility │
│ • Shell usage   │     │ • CGO settings  │     │ • Suggestions   │
└─────────────────┘     └─────────────────┘     └─────────────────┘

Phase 1: Go Code Analysis

Using Go's AST parser, we walk through your source code detecting:

exec.Command and exec.CommandContext calls
Direct CGO imports (import "C")
Shell command invocations (bash, sh, etc.)
System binary dependencies (curl, wget, git, etc.)

We even resolve variables to catch patterns like:

cmd := "bash"
exec.Command(cmd, "-c", "echo hello")  // Still detected!

Phase 2: Dockerfile Analysis

We parse your Dockerfile to understand:

Final base image (scratch, distroless, alpine, debian, etc.)
Multi-stage build configurations
CGO_ENABLED settings
Available binaries in the target environment

Phase 3: Correlation

This is where the magic happens. We cross-reference what your code needs with what your container provides:

Real-World Example: The Security Migration Gone Wrong

Let's walk through a typical scenario. You have a Go service that fetches data from an external API, and security has flagged your container for having too many CVEs:

package main
import (
    "os/exec"
    "log"
)
func fetchData() ([]byte, error) {
    cmd := exec.Command("curl", "-s", "https://api.example.com/data")
    return cmd.Output()
}
func main() {
    data, err := fetchData()
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("Got %d bytes", len(data))
}

Your current Dockerfile uses a full image (lots of CVEs):

# OLD: 200+ CVEs from base image
FROM golang:1.22
WORKDIR /app
COPY . .
RUN go build -o /app/server .
ENTRYPOINT ["/app/server"]

You migrate to a minimal image (scratch, ubi9-micro, or distroless) for fewer CVEs:

FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server .
FROM scratch
COPY --from=builder /app/server /server
ENTRYPOINT ["/server"]

Running go-runtime-compat:

go-runtime-compat analyze --project . --dockerfile ./Dockerfile

Output:

╔══════════════════════════════════════════════════════════════════╗
║              go-runtime-compat Analysis Report                   ║ ╚══════════════════════════════════════════════════════════════════╝
📊 Summary
──────────────────────────────────────────────────────────────────
  Total Findings: 2
  ❌ Errors:      1
  ⚠️  Warnings:    1
❌ Status: FAILED
📋 Findings by Category
──────────────────────────────────────────────────────────────────
  📁 Correlation (1 finding)
  ────────────────────────────────────────────────────────────
  ❌ [CCG101] System binary 'curl' used but may not be available in scratch
     📍 Location: main.go:10
     💡 Suggestion: Install 'curl' in Dockerfile or use a pure Go alternative
  📁 Exec Command (1 finding)
  ────────────────────────────────────────────────────────────
  ⚠️  [CCG003] System binary 'curl' executed via exec.Command
     📍 Location: main.go:10
     💡 Suggestion: Consider using net/http for HTTP requests

Now you have a choice:

Go back to the vulnerable image (not acceptable)
Install curl in the minimal image (adds attack surface back)
Fix the code to not need external binaries (the right answer)

Replace exec.Command("curl", ...) with Go's native net/http package:

func fetchData() ([]byte, error) {
    resp, err := http.Get("https://api.example.com/data")
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    return io.ReadAll(resp.Body)
}

Re-run the analysis: PASSED.

Now you have both: zero CVEs from the base image and working code.

The Rule System

go-runtime-compat uses a comprehensive rule system with the CCG prefix (Container Compatibility for Go):

Go Code Rules (CCG001–019)

Dockerfile Rules (CCG030–049)

Correlation Rules (CCG100–109)

CI/CD Integration

The real power of static analysis is catching issues before they merge. Here's how to integrate go-runtime-compat into your pipeline:

GitHub Actions

name: Container Compatibility Check
on: [push, pull_request]
jobs:
  compat-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.22'
      - name: Install go-runtime-compat
        run: go install github.com/Jonsy13/go-runtime-compat@latest
      - name: Run compatibility check
        run: go-runtime-compat analyze --project . --dockerfile ./Dockerfile --strict

The --strict flag exits non-zero on warnings, not just errors—perfect for enforcing best practices.

JSON Output for Automation

For programmatic consumption (dashboards, custom tooling, AI agents):

go-runtime-compat analyze --project . --dockerfile ./Dockerfile --output json
{
  "findings": [
    {
      "rule_id": "CCG101",
      "category": "correlation",
      "severity": "error",
      "message": "System binary 'curl' used but may not be available in scratch",
      "location": "main.go:10",
      "suggestion": "Install 'curl' in Dockerfile or use a pure Go alternative"
    }
  ],
  "summary": {
    "total_findings": 1,
    "error_count": 1,
    "warning_count": 0,
    "passed": false
  }
}

Best Practices for Secure, Compatible Go Containers

After building this tool and migrating several projects to minimal images, here are the patterns that let you have both security and reliability:

1. Prefer Pure Go Libraries

2. Disable CGO for Minimal Images

FROM golang:1.22 AS builder
ENV CGO_ENABLED=0
RUN go build -ldflags="-s -w" -o /app/server .

# Choose your minimal base:
# FROM scratch           # Zero CVEs, zero packages
# FROM gcr.io/distroless/static  # Near-zero CVEs
FROM registry.access.redhat.com/ubi9-micro  # RHEL-compatible, minimal CVEs
COPY --from=builder /app/server /server

3. If You Need CGO, Match Your libc

# For Alpine targets, build with Alpine (musl)
FROM golang:1.22-alpine AS builder
RUN apk add --no-cache gcc musl-dev
RUN go build -o /app/server .

FROM alpine:3.19
COPY --from=builder /app/server /server
# For UBI/RHEL targets, build with UBI (glibc)
FROM registry.access.redhat.com/ubi9/go-toolset AS builder
WORKDIR /app
COPY . .
RUN go build -o server .

FROM registry.access.redhat.com/ubi9-minimal
COPY --from=builder /app/server /server

4. Audit External Dependencies

Some popular Go libraries use CGO under the hood:

mattn/go-sqlite3 (requires CGO)
Some image processing libraries
Certain crypto implementations

These will break your migration to minimal images. Use go-runtime-compat to detect them before they surprise you.

5. Run Compatibility Checks Before Security Migrations

Before switching base images to reduce CVEs:

# Check if your code is compatible with the new minimal image
go-runtime-compat analyze --project . --dockerfile ./Dockerfile.new --strict

This catches issues during the migration, not after deployment.

The Architecture

For those interested in the internals, go-runtime-compat is built with a clean, extensible architecture:

go-runtime-compat/
├── main.go                          # CLI entry point
└── internal/
    ├── analyzer/
    │   └── go_analyzer.go           # Go AST analysis
    ├── cli/
    │   └── root.go                  # Cobra commands
    ├── correlator/
    │   └── correlator.go            # Cross-reference engine
    ├── docker/
    │   ├── dockerfile_analyzer.go   # Dockerfile parsing
    │   └── image_inspector.go       # Docker image inspection
    ├── report/
    │   └── reporter.go              # Output formatting
    └── rules/
        ├── engine.go                # Rules evaluation
        └── types.go                 # Type definitions

The correlation engine is the heart of the tool. It maps code dependencies to container capabilities:

type Dependency struct {
    Type     DependencyType  // exec_command, shell_command, cgo, etc.
    Name     string          // The binary or library name
    Location rules.Location  // Where it was detected
}
type DockerfileCapability struct {
    HasShell       bool
    HasGlibc       bool
    HasMusl        bool
    IsMinimalImage bool
    // ...
}

When a shell command is detected in code but the Dockerfile uses scratch, the correlator generates a CCG100 error. When CGO is detected but the target is Alpine, it generates CCG104.

What's Next?

I'm actively working on new features:

Kubernetes manifest analysis — Detect issues in deployment configurations
Runtime binary inspection — Use ldd to verify actual linking
Custom rule definitions — Define your own project-specific rules
IDE integration — Real-time feedback in VS Code and GoLand

Try It Today

# Install
go install github.com/Jonsy13/go-runtime-compat@latest

# Analyze your project
go-runtime-compat analyze --project . --dockerfile ./Dockerfile

# See all available rules
go-runtime-compat rules

The tool is open source and available on GitHub. Contributions, issues, and feedback are welcome!

Conclusion

Reducing container vulnerabilities by migrating to minimal base images is the right move for security. But it introduces a new class of compatibility bugs that traditional testing doesn't catch.

go-runtime-compat lets you have both: the security benefits of minimal images and confidence that your code will actually run.

Don't choose between secure and working. Analyze compatibility before you migrate.

If you found this useful, give the repo a star! Questions or feedback? Open an issue or reach out on Twitter.

#golang #docker #containers #security #vulnerability