Architecting a Self-Healing Security Boundary for EKS Ingress

If an ALB exists, it must be protected — automatically, not eventually. A staged, event-driven reference architecture for EKS ingress security, covering what AWS and Gateway API now handle natively, what still doesn't exist, and how to own only the gap that remains.

The problem: the ALB and its security controls follow different lifecycles

When I first examined this problem, I found that creating the Application Load Balancer was not the difficult part. An application team could deploy a Kubernetes Ingress, and the AWS Load Balancer Controller would asynchronously provision an internet-facing ALB.

The security lifecycle was different.

The ALB did not automatically inherit the complete security baseline I needed: an approved AWS WAF WebACL, Shield Advanced protection, a Route 53 health check associated with that protection, central security logging, lifecycle inventory and reliable cleanup when the ALB was deleted.

The timing also mattered. The ALB ARN did not exist when the Kubernetes manifest was submitted. It became available only after the controller created the AWS resource. I therefore could not reliably complete every WAF, Shield and Route 53 operation as part of the same application deployment step.

I saw several independently managed control planes involved:

Kubernetes expressed the application's routing intent.
The AWS Load Balancer Controller owned the ALB lifecycle.
AWS WAF provided the association API, while the platform automation performed the association.
Shield Advanced owned the protection and health-check association.
Route 53 owned DNS records and external health checks.
The security platform owned logging, governance and audit evidence.

Each control plane operated on its own lifecycle. A successful Kubernetes deployment proved that the routing resources had been accepted; it did not prove that the resulting ALB had reached the organisation's required security state.

The usual response was to add another pipeline, an infrastructure-as-code run or a manual remediation step after the ALB appeared. I did not consider that predictable enforcement. It made protection dependent on every application team remembering and successfully executing the next step. It also created opportunities for delayed protection, inconsistent configuration and incomplete cleanup.

Figure 1 — The original EKS ingress security lifecycle gap. The application path creates the internet-facing ALB, while its additional security controls depend on a separate workflow that begins only after the ALB exists.

This led me to the architectural question explored in this article:

how could I make ingress security a lifecycle-driven platform capability — automatically applying, verifying, reconciling, and removing the required controls for every intended internet-facing ALB — without depending on an application team to remember and successfully execute another pipeline step?

Explore the implementation

👉mdevendr/eks-ingress-security-automation

The repository separates the architecture into:

stages/00-ingress — original Ingress and event-driven model
stages/01-gateway-api — deterministic Gateway API implementation
stages/02-kro — optional SecureALB platform abstraction
docs/architecture-evolution.md — platform version timeline and controller responsibilities

Evidence scope: The reference implementation does not demonstrate AWS Shield Advanced end to end because activating the service requires a significant commercial commitment — approximately USD 3,000 per organisation per month, subject to current AWS pricing and terms.

The implementation validates the ALB lifecycle event flow, controller behaviour, DynamoDB inventory, health-check discovery and reconciliation without activating the paid service. The Shield Advanced protection and Route 53 health-check association paths are implemented but require a Shield Advanced-enabled account for complete end-to-end validation.

Stage 0: Immediate Remediation via Event-Driven Automation

My first objective was to close the security lifecycle gap without changing how application teams deployed workloads. They continued to use Kubernetes Ingress, and the AWS Load Balancer Controller continued to create the internet-facing Application Load Balancer.

I added an event-driven security layer around the resulting ALB lifecycle.

When AWS CloudTrail records a CreateLoadBalancer event, Amazon EventBridge invokes a Lambda function. The function inspects the newly created ALB and its tags, determines whether the load balancer is within the automation scope, and applies the required controls.

For an eligible ALB, the automation:

associates the designated AWS WAF WebACL;
creates AWS Shield Advanced protection;
creates a Route 53 health check;
associates that health check with the Shield protection;
creates or updates the Route 53 alias record; and
stores the ALB and dependent-resource identifiers in DynamoDB.

The DynamoDB record provides the lifecycle correlation that is not present in the original CloudTrail event. When CloudTrail later records DeleteLoadBalancer, the Lambda retrieves the stored state and removes the dependent Route 53 health check, DNS record, Shield protection and inventory record.

This made security enforcement a platform response to the actual AWS resource lifecycle. Application teams did not need an additional security deployment, and the automation did not have to predict when the AWS Load Balancer Controller would finish creating the ALB.

Figure 2 — Stage 0 uses CloudTrail, EventBridge and Lambda to detect the lifecycle of ALBs created from Kubernetes Ingress and automatically manage their associated security controls.

Reference — https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/

Stage 0 closed the immediate operational gap: the platform could discover an ALB after creation, apply a consistent security baseline and clean up the associated resources after deletion. It also established the deterministic lifecycle model that the later architecture retains.

Architectural evolution

Stage 0 reflected the platform capabilities available when I originally designed the solution. Kubernetes Ingress and AWS Load Balancer Controller annotations configured the ALB, while the event-driven layer managed the security lifecycle after the ALB was created.

As Kubernetes, Amazon EKS and the surrounding controller ecosystem evolved, I revisited where each responsibility should reside. My objective was to move supported operations to purpose-built controllers while retaining event-driven automation for the remaining cross-service lifecycle gap.

Figure 3 — The milestones that enabled the architecture to evolve from annotation-driven Ingress automation to a controller-reconciled Gateway API model and an optional platform API.

From bespoke automation to controller reconciliation

The general availability of the first ACK service controllers in April 2022 established a model in which supported AWS resources could be declared as Kubernetes custom resources and continuously reconciled by service-specific controllers. This later allowed the Route 53 health check to be represented as an ACK HealthCheck instead of being created and deleted by Lambda.

Gateway API v1.0 became generally available in October 2023. Its stable GatewayClass, Gateway and HTTPRoute APIs provided a clearer, role-oriented alternative to the annotation-heavy Ingress model. Gateway API v1.0 announcement

ValidatingAdmissionPolicy subsequently became generally available in Kubernetes 1.30, followed by Amazon EKS 1.30 in May 2024. This gave me a stable native mechanism for rejecting insecure configurations before a controller created or modified AWS resources. I intentionally do not apply this capability retrospectively to the pre-EKS-1.30 Stage 0 design. Kubernetes admission-policy GA

Stage 1: Moving security intent into Gateway API

The first evolution replaces the annotation-heavy Ingress model with explicit Gateway API and controller-specific resources.

I deploy the following resources together:

Gateway defines the internet-facing entry point and references its LoadBalancerConfiguration.
HTTPRoute defines the hostname, routing rules and backend service.
LoadBalancerConfiguration defines ALB-level settings, including the WebACL and Shield Advanced protection.
TargetGroupConfiguration defines target registration and health behaviour.
ACK HealthCheck declares the Route 53 health-check endpoint.

Figure 4 — Stage 1 expresses the ALB, routing, security and health-check intent through Gateway API and controller-specific Kubernetes resources.

apiVersion: gateway.k8s.aws/v1beta1
kind: LoadBalancerConfiguration
metadata:
  name: platform-baseline
  namespace: platform-gateways
spec:
  scheme: internet-facing
  ipAddressType: ipv4
  defaultTargetGroupConfiguration:
    name: demo-targets
  wafV2:
    webACL: __WEB_ACL_ARN__
  shieldConfiguration:
    enabled: true
  loadBalancerAttributes:
    - key: deletion_protection.enabled
      value: "false"
    - key: routing.http.drop_invalid_header_fields.enabled
      value: "true"
    - key: access_logs.s3.enabled
      value: "true"
    - key: access_logs.s3.bucket
      value: __ACCESS_LOG_BUCKET__
    - key: access_logs.s3.prefix
      value: secure-alb/stage-1
  tags:
    secure-alb/id: secure-alb-demo/demo
    secure-alb/web-acl-arn: __WEB_ACL_ARN__
    secure-alb/stage: gateway-api-http-evidence

apiVersion: gateway.k8s.aws/v1beta1
kind: LoadBalancerConfiguration
metadata:
  name: platform-baseline
  namespace: platform-gateways
spec:
  scheme: internet-facing
  ipAddressType: ipv4
  defaultTargetGroupConfiguration:
    name: demo-targets
  wafV2:
    webACL: __WEB_ACL_ARN__
  shieldConfiguration:
    enabled: true
  loadBalancerAttributes:
    - key: deletion_protection.enabled
      value: "false"
    - key: routing.http.drop_invalid_header_fields.enabled
      value: "true"
    - key: access_logs.s3.enabled
      value: "true"
    - key: access_logs.s3.bucket
      value: __ACCESS_LOG_BUCKET__
    - key: access_logs.s3.prefix
      value: secure-alb/stage-1
  tags:
    secure-alb/id: secure-alb-demo/demo
    secure-alb/web-acl-arn: __WEB_ACL_ARN__
    secure-alb/stage: gateway-api-http-evidence

Controller ownership

Each controller watches only the resources within its responsibility:

AWS Load Balancer Controller creates the ALB, listener and target groups. It also associates the WebACL and creates the Shield Advanced protection from LoadBalancerConfiguration.
ExternalDNS reads the hostname from HTTPRoute and the ALB target from the accepted Gateway status, then manages the Route 53 alias record.
ACK Route 53 controller watches the HealthCheck custom resource and creates the corresponding Route 53 health check using the expected FQDN.
ValidatingAdmissionPolicy rejects a LoadBalancerConfiguration that does not meet the platform's security requirements.

The ACK controller does not watch HTTPRoute or discover the ALB. The hostname is supplied independently to both HTTPRoute and the ACK HealthCheck, allowing the health check to be provisioned before the ALB and DNS alias exist.

Route 53 initially considers a newly created health check healthy until it has accumulated enough observations to determine its actual status. As ExternalDNS and the AWS Load Balancer Controller converge and the endpoint becomes reachable, Route 53 health-checker probes determine the reported application health.

The remaining event-driven responsibility

The controllers remove most of the bespoke provisioning logic, but no controller owns the relationship between the ACK-created health check and the Shield protection created by AWS Load Balancer Controller.

The CloudTrail and EventBridge path therefore remains, but with a narrower role. The Lambda:

Discovers the ALB and its Shield protection.
Locates the corresponding Route 53 health check.
Associates or disassociates the health check with Shield.
Stores the lifecycle correlation in DynamoDB.

A scheduled reconciler compares the expected and actual state and repairs missed associations or drift.

Stage 1 is therefore not the removal of event-driven automation. It is a redistribution of responsibility: controller-native reconciliation performs the primary provisioning, while the event-driven layer provides cross-service lifecycle assurance.

Architectural note: Multi-account scope and failure resilience

In a multi-account AWS Organization, I would separate central policy governance from workload-local lifecycle assurance. AWS Firewall Manager can apply WAF and Shield Advanced policies across organisational units and centralise WAF logging, while the protected ALBs remain in spoke application accounts. The remaining integration is resource-specific: correlating each ALB protection with the Route 53 health check declared by its EKS workload. I can deploy the assurance component in each spoke or centralise it through tightly scoped STS AssumeRole permissions.

The health-check association is operationally significant. Route 53 initially considers a newly created health check healthy until it has enough observations to determine its actual status.

Because the initial healthy state is provisional, a Shield-enabled production workflow should confirm that the health check is actively passing against the application endpoint before associating it with Shield Advanced. This establishes a reliable health signal for Shield Advanced health-based DDoS detection. Once associated, Shield Advanced uses application health to improve detection accuracy and responsiveness and can mitigate at lower thresholds when the endpoint becomes unhealthy. A healthy associated Route 53 health check is also required for SRT proactive-engagement support.

The implementation uses adaptive AWS SDK retries, dead-letter queues, check-before-associate logic, DynamoDB lifecycle inventory and scheduled reconciliation. For a production multi-account deployment, I would extend this with conditional DynamoDB writes, explicit lifecycle states, bounded retry orchestration and per-account failure isolation.

AWS Firewall Manager

Where AWS Firewall Manager fits

AWS Firewall Manager provides an optional organisation-wide governance layer. From a delegated administrator account, a security team can apply AWS WAF and Shield Advanced policies across selected accounts and organisational units, automatically bring in-scope ALBs under policy, and configure central WAF logging to Amazon Data Firehose or Amazon S3 for onward SIEM ingestion.

That makes Firewall Manager a strong fit where central security owns a common baseline. It is not mandatory for this design, however, and it does not replace the workload-local correlation between an ALB, its Shield protection and the ACK-created Route 53 health check.

I therefore separate the scopes: Firewall Manager can govern organisation-wide WAF, Shield and logging policy, while the event-driven assurance layer maintains per-ALB lifecycle inventory and the Shield health-check association. This separation works with either Stage 1 or the optional Stage 2 abstraction.

Where Firewall Manager stops in this design

Firewall Manager does not create the workload-specific ACK HealthCheck resource or correlate it with the Shield protection created for that ALB. Its compliance view is policy- and resource-oriented rather than the cross-controller, per-deployment lifecycle record held in DynamoDB.

I do not treat central administration as a product weakness; it is an operating-model choice. Likewise, I do not attribute SIEM routing or SRT enablement to the Lambda: Firewall Manager can centralise WAF logs, while Shield Response Team contact and proactive-engagement setup remain account-level operational prerequisites. The narrow gap left for this architecture is the health-check correlation and its continuous verification.

NOTE: Firewall Manager doesn't support Amazon Route 53 or AWS Global Accelerator. If you need to protect these resources with Shield Advanced, you can't use a Firewall Manager policy. Instead, follow the instructions in Adding AWS Shield Advanced protection to AWS resources.

👉— https://docs.aws.amazon.com/waf/latest/developerguide/getting-started-fms-shield.html

Stage 2: Simplifying consumption with kro

Stage 1 gives me a deterministic Gateway API implementation, but it also exposes several resources to application teams. They need to understand Gateway, HTTPRoute, LoadBalancerConfiguration, TargetGroupConfiguration and the ACK HealthCheck.

I wanted to simplify that experience without changing the proven architecture underneath.

With kro, my platform team defines a ResourceGraphDefinition once. This registers a purpose-built SecureALB API. An application team then provides only its hostname, service, port, certificate and a logical WAF policy such as regulated-public.

When the team creates a SecureALB instance, kro generates the complete Stage 1 resource graph. The WAF policy is resolved through a platform-controlled catalogue, so the application never supplies a raw WebACL ARN.

Figure 5 — kro packages the proven Gateway API and ACK resources behind a platform-authored SecureALB API.

What changes here is the application experience — not the security implementation. AWS Load Balancer Controller still manages the ALB, WebACL association and Shield protection. ExternalDNS still manages the alias, ACK still creates the health check, and the event-driven layer still handles the Shield health-check association and lifecycle inventory.

kro is therefore not my security boundary. RBAC controls who can create SecureALB resources, and an independent admission policy validates the generated configuration.

Stage 2 is optional. I would use it when I want to offer ingress security as a simple, governed platform capability. If an organisation prefers direct ownership of Gateway API resources, Stage 1 remains a complete current architecture.

Recommended evolved architecture

I retain two deterministic layers. The native/controller layer owns what it can reconcile directly: AWS Load Balancer Controller manages the ALB, WebACL association and Shield protection; ExternalDNS manages the alias; and ACK manages the Route 53 health check. Firewall Manager can provide the WAF, Shield and logging baseline instead where organisation-wide central governance is preferred.

The slim event-driven layer owns only the unresolved relationship: it discovers the ALB, Shield protection and ACK health check, associates or disassociates the health check, and records lifecycle state in DynamoDB. Security logging remains with the security platform or Firewall Manager, not with this Lambda.

A practical adoption path

Phase 0 — Preserve Stage 0 for existing Ingress workloads; it remains a valid lifecycle-enforcement model.

Phase 1 — Use the Stage 1 Gateway API model for new workloads and remove WAF, Shield, DNS and health-check creation from the Lambda where controllers now own them.

Phase 2 — Evaluate Firewall Manager where an organisation wants a delegated, OU-wide WAF and Shield baseline; keep spoke-local EDA for the workload-specific health-check relationship.

Phase 3 — Optionally introduce the kro SecureALB API to simplify consumption. This changes the application interface, not the underlying security ownership.

Drift detection and reconciliation

CloudTrail event delivery is best effort, controller reconciliation can fail, and operators can change ALB security associations manually. I therefore use the event-driven path for rapid enforcement and a scheduled reconciler as a safety net. In the reference implementation, the reconciler runs every five minutes, reads the ALB lifecycle inventory from DynamoDB, and checks whether each ALB still has its expected WebACL association, Shield Advanced protection and Shield-associated Route 53 health check.

The reconciler operates in audit mode by default. When explicitly enabled in remediation mode, it can restore a drifted WebACL association and reassociate an existing Route 53 health check with the Shield protection. Missing ALBs, Shield protections or Route 53 health checks are recorded as findings rather than recreated outside their owning controller.

Findings and actions are written to DynamoDB and application logs, which a production implementation can route to the SOC. This reconciliation is limited to ALB lifecycle and security-control attachment drift. It does not inspect or modify WAF rule content, which remains the responsibility of the application-security or central security-policy team.

Measured Outcomes — Original Delivery Model, Controller-Native Model, and EDA

The primary success measure is how quickly a newly created internet-facing ALB receives its intended security controls. In the previous pipeline-driven implementation, this could take up to two hours after an EKS Ingress deployment. In the event-driven implementation protection was typically applied in approximately ten seconds and consistently in under one minute.

These figures are observed implementation results, not AWS service-level guarantees

[Embedded content: 5dccd46950e4afe8a4312a3464a0693b]

The EDA's principal outcome is the reduction of the ALB protection window from hours to seconds without requiring application teams to run another pipeline or remember a follow-up security step.

The approximately ten-second result describes the observed protection latency in the implementation. CloudTrail event delivery remains best effort, so this should not be interpreted as an AWS guarantee. Scheduled reconciliation provides a separate safety net for missed events and subsequent ALB security-association drift.

SRT Proactive Engagement is intentionally excluded as a direct EDA outcome. Associating the Route 53 health check with Shield Advanced enables health-based detection and satisfies an important prerequisite, but SRT authorization, contacts and proactive-engagement configuration remain separate account-level operational responsibilities.

Closing the lifecycle gap

I started with a simple problem: an internet-facing ALB could exist before its complete security baseline was applied.

Stage 0 closed that gap through event-driven automation around Kubernetes Ingress. As the platform evolved, Stage 1 moved supported responsibilities to purpose-built controllers using Gateway API, ExternalDNS and ACK. The event-driven layer became a focused assurance mechanism for the remaining cross-service lifecycle association.

Stage 2 optionally packages the same proven resource graph behind a platform-authored SecureALB API. It simplifies consumption without replacing the deterministic controllers or making kro the security boundary.

The architecture has evolved, but the principle remains unchanged: application teams declare their workload intent, while the platform consistently applies, reconciles and removes the required ingress security controls.

For me, that is the real meaning of secure by design — security is not a follow-up activity that someone must remember. It is part of the resource lifecycle itself.

Evidence context: This comparison reflects the capabilities and operational outcomes of the original enterprise EDA implementation, including Shield Advanced, SRT readiness and central SIEM integration. The GitHub repository is a lower-cost reference implementation of the architectural pattern and does not demonstrate every Shield Advanced, SRT and SIEM capability end to end.

Reference — https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/4042

Future extension: GenAI-assisted security operations

I would not place generative AI in the ALB creation path or invoke it for every CloudTrail event. ALB protection must remain deterministic, fast and predictable through controllers, EventBridge, Lambda and reconciliation.

The potential value of GenAI sits above this proven control plane as an operational-assistance layer. It could periodically analyse ALB lifecycle findings, reconciliation results, WAF and Shield telemetry, and security-platform alerts to:

Summarise the security posture of internet-facing ALBs across accounts and clusters.
Correlate repeated reconciliation failures with related AWS service or controller events.
Explain why an ALB is non-compliant and identify the responsible control plane.
Produce incident timelines and evidence summaries for security and audit teams.
Recommend remediation actions using approved operational runbooks.
Prioritise findings according to workload criticality, exposure and business context.

The AI layer would not directly create, delete or modify security controls. Any recommendation would be converted into a constrained, deterministic action and validated against platform policy before execution. High-impact actions would require human approval.

Inputs would also be treated as untrusted data. Request headers, user agents, WAF logs and other attacker-controlled fields would be normalised before being supplied to the model to prevent security telemetry from becoming an instruction channel.

This remains a future operational extension rather than Stage 3 of the implemented architecture. I would introduce it only when the deterministic stages have produced enough reliable evidence to justify the additional reasoning cost, governance controls and operational complexity.

About me

I'm a Cloud Architect specialising in secure, scalable and identity-driven platforms across AWS, Azure and Google Cloud. My work spans agentic AI, zero-trust access patterns, serverless architectures, resilient distributed systems and regulated workloads where security, correctness and architectural clarity matter most.

Connect with me on LinkedIn or read more of my work on Medium.