Post cover image
Photo by Dave Lowe on Unsplash

July 29, 2026

We Made Our Access Control Too Granular. Here’s What It Cost Us

One environment, one app, one role — sounds like best practice. After a maintenance cycle that touched hundreds of roles across GCP…

By Sriram Mahalingam

6 min read

PLATFORM ENGINEERING & SECURITY

One environment, one app, one role — sounds like best practice. After a maintenance cycle that touched hundreds of roles across GCP, Kubernetes, and Spring Security, I'm not so sure anymore

friendly link here

None

There is a version of security best practice that sounds unimpeachable in a design review and becomes a maintenance nightmare in production. Least privilege access is one of those principles. Nobody argues against it. The idea is correct. The implementation, if you take it too literally, can quietly become one of the most expensive decisions your platform team ever made.

We learned this the hard way during a recent access control maintenance cycle. What started as a self-attestation exercise — reviewing who had access to what and confirming it was still needed — turned into a multi-week project that touched hundreds of roles across three layers of our stack. GCP IAM, Kubernetes RBAC, and Spring Security. One environment, one application, one role. Every combination.

The principle was sound. The execution was overkill. Here is what we found

How We Got Here

The journey to over-granular access control rarely happens in one decision. It happens in increments, each one individually defensible.

It starts with a reasonable observation: production access should be different from dev access. Correct. Then someone points out that read access should be different from write access. Also correct. Then a security audit flags that service accounts are shared across applications. Fair point. Then compliance asks for role-per-application separation. Understandable. Then someone extends the same logic to Kubernetes namespaces. Then to Spring Security method-level annotations.

Each decision made sense in isolation. Together they produced a role matrix that looked like this:

Environment (dev/staging/prod) ×
Application (30+ microservices) ×
Role (read/write/admin/deploy/monitor)

= 450+ distinct role combinations
  across GCP IAM alone

Add Kubernetes RBAC:
Namespace × ServiceAccount × ClusterRole
= Another 200+ bindings

Add Spring Security:
Per-service method-level annotations
= Distributed across 30 codebases

When we sat down for the self-attestation exercise — confirming that every role assignment was still valid and still needed — we were looking at hundreds of individual access grants that needed human review. Not automated review. Human review. Someone had to confirm that the payments-service-staging-read role assigned to the reporting-team-sa service account was still legitimate and still needed.

The Three Layers of Pain

GCP IAM granularity feels clean when you define it. You create custom roles scoped to specific resources, bind them to specific service accounts, and tell yourself that you have achieved precise least-privilege access.

What you have actually achieved is a role catalogue that grows multiplicatively with every new application and every new environment.

# What fine-grained GCP IAM looks like in Terraform
# Multiply this by 30 applications × 3 environments

resource "google_project_iam_custom_role" "payments_staging_reader" {
  role_id     = "payments_service_staging_reader"
  title       = "Payments Service Staging Reader"
  permissions = [
    "pubsub.subscriptions.consume",
    "pubsub.topics.publish",
    "cloudtrace.traces.patch",
    "logging.logEntries.create"
  ]
}

resource "google_project_iam_member" "payments_staging_reader_binding" {
  project = var.project_id
  role    = google_project_iam_custom_role.payments_staging_reader.id
  member  = "serviceAccount:payments-staging@${var.project_id}.iam.gserviceaccount.com"
}

Individually, each role definition is clean. At scale, maintaining them — updating permissions when an API changes, rotating service accounts, tracking which role covers which capability — becomes a full-time job nobody was hired to do.

The self-attestation exercise revealed something uncomfortable: several roles had permissions that were added for a specific incident two years ago and never removed. Nobody remembered why they were there. The over-granularity that was supposed to make access precise had instead made it opaque.

Kubernetes RBAC — The Namespace Proliferation Problem

Kubernetes RBAC encourages namespace-level isolation, and the right instinct is to scope permissions to the namespace where the workload runs. The problem is that when you have thirty microservices across three environments and you are scoping service account permissions per application per namespace, the number of bindings multiplies fast.

# Clean in theory — one ServiceAccount per application per namespace
# Multiply by 30 apps × 3 environments = 90 ServiceAccounts
# Each with its own RoleBinding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-service-sa
  namespace: payments-staging

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-service-rolebinding
  namespace: payments-staging
subjects:
  - kind: ServiceAccount
    name: payments-service-sa
    namespace: payments-staging
roleRef:
  kind: Role
  apiGroup: rbac.authorization.k8s.io
  name: payments-service-role

The maintenance cost is not just the YAML. It is the cognitive overhead of tracking which ServiceAccount belongs to which application in which namespace. When a developer raises a ticket saying their service cannot read from a ConfigMap, diagnosing which of the ninety ServiceAccount bindings is missing the right permission requires tracing through multiple layers of RBAC rules that are scattered across namespace-scoped Roles and cluster-scoped ClusterRoles.

During the self-attestation exercise, several teams could not confirm what their service accounts actually needed because the person who had originally set them up had left, and the configuration had never been documented beyond the YAML itself.

Spring Security — The Distributed Configuration Problem

Application-level RBAC through Spring Security method annotations is the layer where over-granularity becomes hardest to audit because it is distributed across every service's codebase rather than centralised in a single place.

// What granular Spring Security looks like
// Multiply across 30 services, each with its own role definitions

@RestController
@RequestMapping("/api/payments")
public class PaymentController {

    @GetMapping("/{id}")
    @PreAuthorize("hasRole('PAYMENTS_READ') or hasRole('PAYMENTS_ADMIN')")
    public ResponseEntity<Payment> getPayment(@PathVariable String id) {
        return ResponseEntity.ok(paymentService.findById(id));
    }

    @PostMapping
    @PreAuthorize("hasRole('PAYMENTS_WRITE') or hasRole('PAYMENTS_ADMIN')")
    public ResponseEntity<Payment> createPayment(@RequestBody PaymentRequest request) {
        return ResponseEntity.ok(paymentService.create(request));
    }

    @DeleteMapping("/{id}")
    @PreAuthorize("hasRole('PAYMENTS_ADMIN')")
    public ResponseEntity<Void> deletePayment(@PathVariable String id) {
        paymentService.delete(id);
        return ResponseEntity.noContent().build();
    }
}

Per-service, per-operation role definitions give you fine control. Across thirty services, you have thirty different role naming conventions, thirty different interpretations of what "admin" means, and no centralised view of what a given user or service account can actually do across the platform.

Compliance asked us for an audit of which service accounts had write access across which services. Answering that question required opening thirty codebases and reading the [@PreAuthorize](http://twitter.com/PreAuthorize) annotations individually. There was no central registry. The granularity that was supposed to give us precision gave us instead a distributed, unqueryable access control system.

What the Self-Attestation Exercise Actually Revealed

The self-attestation process was supposed to confirm that all access was still valid. What it actually revealed was that over-granular access control has a specific failure mode that coarser access control does not: it becomes unmaintainable fast enough that nobody really knows what access exists anymore.

With coarser roles — environment-level access rather than application-level access — a human can reason about the access model. "This service account has read access to staging." Clear. Auditable. The person doing the self-attestation can confirm it is still needed.

With fine-grained roles, the access model becomes too complex for human reasoning at scale. The payments-service-staging-pubsub-subscriber-v2 role is precise. But nobody can hold the full picture of four hundred and fifty similarly precise roles in their head simultaneously. The self-attestation becomes a box-ticking exercise rather than a genuine security review, because the complexity defeats the human attention that security reviews depend on.

Security that is too complex to understand is not more secure than security that is simpler. It is security that has traded comprehensibility for the appearance of precision.

What We'd Do Differently

The answer is not to abandon least privilege. It is to apply it at the right level of granularity — the level where the access model remains comprehensible to the humans who have to maintain and audit it.

For GCP IAM, environment-level custom roles rather than application-level ones:

# Coarser but maintainable — per environment, not per app
# 3 roles instead of 90

resource "google_project_iam_custom_role" "platform_staging_operator" {
  role_id     = "platform_staging_operator"
  title       = "Platform Staging Operator"
  permissions = [
    "pubsub.subscriptions.consume",
    "pubsub.topics.publish",
    "cloudtrace.traces.patch",
    "logging.logEntries.create",
    "secretmanager.versions.access"
  ]
}

For Kubernetes RBAC, namespace-scoped roles shared across applications within the same namespace rather than per-application roles:

# Shared Role within namespace — 3 roles per environment
# Not 30 roles per environment

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workload-standard-role
  namespace: staging
rules:
  - apiGroups: [""]
    resources: ["configmaps", "secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]

For Spring Security, centralised role definitions at the platform level rather than per-service definitions:

// Platform-level roles defined once, used everywhere
// Not reinvented per service

public enum PlatformRole {
    READER,      // read access across all services
    WRITER,      // write access across all services
    OPERATOR,    // operational access — restart, redeploy
    ADMIN        // full access
}

// Services use platform roles, not their own inventions
@PreAuthorize("hasRole('WRITER') or hasRole('ADMIN')")
public ResponseEntity<Payment> createPayment(@RequestBody PaymentRequest request) {
    return ResponseEntity.ok(paymentService.create(request));
}

The principle is the same across all three layers: granularity should be fine enough to enforce meaningful boundaries and coarse enough that humans can reason about the access model without a spreadsheet.

The Principle Behind the Principle

Least privilege access is the right principle. The implementation question is: least privilege at what level of granularity?

The answer depends on what you are actually protecting. If you have applications with genuinely different risk profiles — a payments service and a logging service sitting in the same namespace — then application-level separation is worth the maintenance cost. If your thirty microservices are all broadly similar in risk and all serving the same platform, environment-level separation is probably sufficient and significantly cheaper to maintain.

The mistake we made was applying maximum granularity uniformly, because maximum granularity sounds like maximum security. It does not produce maximum security. It produces maximum complexity, and complexity is the enemy of the auditability that security depends on.

The best access control model is the one your team can actually understand, maintain, and audit — not the one that looks most impressive in a design review.