Beyond the Pipeline: Securing Kubernetes at Runtime with DevSecOps

Introduction

In the previous article, I explored how automated security checks in a GitHub Actions pipeline help identify vulnerabilities before software reaches production.

Secret scanning, static analysis, dependency scanning, infrastructure validation, and dynamic testing all contribute to reducing risk during development.

However, passing every security check does not guarantee that an application will remain secure after deployment, containers can be compromised, secrets can be exposed, attackers can exploit vulnerabilities that were not identified during testing, infrastructure can drift from its intended configuration. Modern DevSecOps recognizes that deployment is not the end of security. It is the beginning of continuous verification.

This is where runtime security becomes essential.

Why Runtime Security Matters

Imagine a pipeline that completes successfully, every scanner reports green, the container image is signed, deployment succeeds. A week later, an attacker exploits a newly disclosed vulnerability in a dependency that was unknown during the build process.

The pipeline did everything it was designed to do, the application is still compromised, this illustrates an important lesson, build-time security reduces risk. Runtime security detects and responds to threats that emerge after deployment.

Both are necessary.

Runtime Security in the Project

The project architecture included several runtime security controls that continued protecting the application after deployment.

These included:

• Falco
• HashiCorp Vault
• Kubernetes Network Policies
• OPA Gatekeeper
• Observability

Together, these controls provided continuous visibility and policy enforcement inside the Kubernetes environment.

Figure 1. Runtime security controls within the SecureFlow architecture. While the CI/CD pipeline helps prevent vulnerable software from reaching production, components such as OPA Gatekeeper, HashiCorp Vault, Falco, Network Policies, and the observability stack continue protecting workloads after deployment.

Detecting Suspicious Activity with Falco

Applications are not the only things running inside containers.

Processes execute, files change, network connections are created.

Attackers often leave traces through unusual runtime behavior.

Falco continuously monitors container activity and compares events against predefined security rules.

Examples include:

• Unexpected shell execution inside containers
• Privilege escalation attempts
• Unauthorized file modifications
• Suspicious process execution
• Unexpected outbound connections

Rather than preventing attacks, Falco focuses on detecting abnormal behavior quickly enough for security teams to respond.

This represents the "Verify Right" philosophy in action.

Figure 2. Falco detecting suspicious activity within a Kubernetes workload.

Protecting Secrets with HashiCorp Vault

• Applications require secrets.
• Database credentials.
• API keys.
• JWT signing keys.
• Encryption keys.

Storing these directly inside source code or Kubernetes manifests introduces unnecessary risk. Instead, the implementation uses HashiCorp Vault as a centralized secrets management platform. Applications retrieve secrets only when needed.

Developers never need to embed sensitive credentials within repositories.

This reduces the impact of accidental secret exposure while improving credential management.

Restricting Communication with Network Policies

One compromised container should not automatically expose the rest of the cluster.

Kubernetes Network Policies reduce this risk by controlling which workloads are allowed to communicate.

The project follows a default deny approach.

Services communicate only with explicitly authorized components.

For example:

• The frontend communicates with the authentication service.
• The authentication service communicates with its database.
• The transaction service communicates with the transaction database.
• Unnecessary communication paths remain blocked.

This limits lateral movement if an attacker compromises a workload.

Figure 3. Kubernetes Network Policies restricting communication between services.

Enforcing Security Policies with OPA Gatekeeper

Runtime security also includes preventing insecure workloads from entering the cluster.

OPA Gatekeeper provides policy enforcement during deployment.

Examples include rejecting workloads that:

• Run privileged containers
• Use the latest image tag
• Omit resource limits
• Lack required labels
• Violate organizational security policies

Instead of relying on manual reviews, these policies are enforced automatically.

Policy as Code makes security consistent across deployments.

Visibility Through Observability

Security teams cannot protect systems they cannot observe.

The architecture includes an observability stack built with:

• Prometheus
• Grafana
• Loki

Prometheus collects operational metrics.

Grafana visualizes those metrics through dashboards.

Loki centralizes logs from Kubernetes workloads.

Together, these components provide the visibility needed to investigate incidents, identify anomalies, and understand the health of the environment.

Alerting integrations ensure that important events are communicated quickly.

Figure 4. Monitoring and logging architecture supporting runtime visibility.

Bringing Everything Together

One of the biggest lessons from this implementation was realizing that security is layered. Threat modeling identified risks. The CI/CD pipeline reduced those risks before deployment.

Runtime security continuously validated that protections remained effective afterward.

Each layer supports the others.

Removing any one of them creates blind spots.

Lessons Learned

Before working on this project, I viewed security primarily as something that happened during development. Implementing runtime controls changed that perspective. Applications continue to evolve after deployment.

Attackers continue searching for weaknesses. New vulnerabilities continue to be discovered. Security must therefore continue operating throughout the application's lifetime.

This is the practical meaning of "Verify Right." It is not a single security tool. It is an ongoing commitment to validating that deployed applications remain secure.

Conclusion

Runtime security completes the DevSecOps lifecycle. Threat modeling identifies risk during design. CI/CD pipelines reduce risk before deployment.

Runtime controls monitor, enforce, and validate security after deployment.

Together, these practices create a continuous security model that extends from design through production. Building secure software is not about preventing every attack. It is about continuously reducing risk, detecting abnormal behavior, and responding effectively when threats emerge.

Resources

• Part 1: Shift Left, Verify Right: Building Security That Doesn't Slow Down Development
• Part 2: Implementing Shift Left, Verify Right in a Vulnerable FinTech Microservices Application
• Part 3: Threat Modeling a FinTech Microservices Application: Where DevSecOps Really Begins
• Part 4: Inside a DevSecOps CI/CD Pipeline: How Automated Security Checks Protect Every Commit
• GitHub Repository: https://github.com/Lhilove/SecureFlow-DevSecOps

Next Up

This article explored how runtime security extends DevSecOps beyond the CI/CD pipeline through continuous monitoring, policy enforcement, secrets management, and observability.

In the final article of this series, I'll reflect on the lessons I learned while implementing this DevSecOps security architecture. I'll share how my understanding of secure software delivery evolved, the challenges I encountered, and the principles I'll carry forward into future Application Security and DevSecOps projects.

Thank you for following along with this series. I hope it has shown that effective DevSecOps is not about adopting more security tools, but about integrating security thoughtfully into every stage of the software development lifecycle.