When the Network Fails, Your Control Plane Must Survive

On 4 October 2021, Meta issued a command "with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network."

An audit tool that should have stopped it had a bug. The DNS servers, doing exactly what they were built to do, withdrew their own route advertisements — "our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection." The result was precise and awful: "our DNS servers became unreachable even though they were still operational."

Then comes the sentence this article is about:

"it was not possible to access our data centers through our normal means because their networks were down, and second, the total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this"

Their recovery tools depended on the thing that was broken.

Nobody designs that. It is not a decision anyone makes. It is a property of a graph nobody drew.

Three planes, and only one of them gets designed

Most estates have three distinct paths and a name for only one of them.

Your data plane serves customers. It gets load tested, capacity planned, and put on a dashboard.

Your control plane is how operators change the data plane: the console, the CI pipeline, the bastion, the admin API. It gets built incrementally by whoever needed it, and it inherits every dependency of everything it touches.

Your recovery plane is how operators reach the system when the control plane is gone. Almost nowhere is it a designed artifact. It is whatever happens to still work, discovered at the worst possible moment.

This third plane is the subject here, and the claim is narrow: you cannot know whether you have one without drawing the dependency graph, because the answer is not intuitable from the component list.

A differently named tool is not an independent tool

Here is the trap, and it catches careful people.

Suppose your runbooks are in the wiki. The wiki authenticates through the corporate identity provider and resolves through corporate DNS. Your incident channel is a different product from a different vendor — and it also authenticates through that identity provider and resolves through that DNS.

Two tools. One failure domain. What matters is not which product but which dependencies, and product diversity actively disguises dependency concentration. Three vendors all federating through one identity provider is one dependency wearing three logos.

Hence a recovery plane has to be derived rather than assembled. Nobody assembles a circular recovery path on purpose; everybody assembles one by adding sensible tools that happen to share a foundation.

Drawing the graph

The companion lab does exactly one thing: it takes a declared dependency graph and a set of recovery steps, and flags every step that depends on the domain it is meant to repair.

Declare components with their failure domains and dependencies. Declare recovery steps with the domain each repairs. Then walk the transitive closure.

Against a fictional estate that looks like most estates:

Step set                    Findings  Safe steps
--------------------------  --------  ----------
Undesigned                        13           0
Designed against the graph         0           3

Step set                    Findings  Safe steps
--------------------------  --------  ----------
Undesigned                        13           0
Designed against the graph         0           3

Simulated result.

The undesigned restore-dns step fails for a reason worth reading aloud, because it is the Meta shape reproduced in a graph:

restore-dns  repairs: network
  → bastion            (fails in: network)
  → bastion → corp-network
  → wiki-runbooks → corp-dns

restore-dns  repairs: network
  → bastion            (fails in: network)
  → bastion → corp-network
  → wiki-runbooks → corp-dns

It reaches its bastion through IAP. IAP needs DNS. So the step cannot run at the only time anyone would ever run it.

The transitive walk is the whole point

Negative control N1 measures why a shallower check is not enough.

Inspecting only direct dependencies finds some problems: the bastion is itself in the network domain, so a direct check catches it. What a direct check misses is every multi-hop chain — wiki-runbooks → corp-dns, admin-console → iap, secret-manager → identity-provider.

Those are the ones that matter, because those are the ones a human reviewer also misses. Nobody looks at "restore DNS using the runbook wiki" and thinks about what the wiki resolves through. The chains are two hops long and completely invisible in a list of tools.

Independence cannot be asserted

An earlier version of the validator had a flag: independent: true, meaning this component keeps working when everything else does not. It survived about ten minutes.

A test broke it immediately. The flag was an assertion, and asserting independence silently excluded that component's entire dependency subtree from validation. Mark the bastion independent and the validator stops looking — which is exactly the failure the tool exists to catch, reintroduced as a feature.

So independence is now derived. A component is an anchor when nothing in its transitive closure belongs to any failure domain, computed from the graph rather than declared by whoever added the row.

That change matters beyond tidiness, because the human version of the same mistake is what produces circular recovery planes in the first place. Someone names a thing "emergency console" and the name becomes the argument. N2 puts it plainly: two differently-named tools, both resting on corp-dns, and neither is an anchor however confidently it is labelled.

What actually survives

If independence must be derived, the question becomes what can sit at the bottom of the graph and genuinely have no dependencies.

An immutable audit log. This one is unusual enough to be worth quoting, because almost nothing else in cloud infrastructure makes this claim. Admin Activity audit logs "are always written; you can't configure, exclude, or disable them." And against the obvious attempt to remove them: "Even if you disable the Cloud Logging API, Admin Activity audit logs are still generated." Plus "Log entries written by Cloud Audit Logs are immutable."

Read that as a threat model rather than a feature list. Most controls protect you from an outsider. That one holds against the operator, which is the only threat model that matters for a path whose entire purpose is elevated access under pressure.

A hardware token. Not a push notification. A push needs a phone, which needs a network, which may be the thing that is broken. The distinction is the same one as the wiki and the chat channel: what does it depend on, not what is it called.

A printed runbook. The lab ships one, and its first paragraph explains why it is printed rather than apologising for it. A runbook in the wiki depends on the wiki. The validator flags restore-dns → wiki-runbooks → corp-dns, and during the failure that runbook exists for, it is unreachable.

A separate identity path. This is the hard one, and the place most recovery planes quietly fail. An identity path that federates through the primary identity provider is not a recovery path. N3 demonstrates the fragility: take the designed step set, which has zero findings, and move one edge — put the out-of-band console behind the primary identity provider — and the entire recovery plane becomes invalid. One edge.

Break glass, and the four rules

Emergency access is where good intentions produce the worst artifacts. The instinct after reading an outage report is to make emergency access harder, and taken far enough that produces a path nobody can use in an emergency, which is the same as not having one.

Privileged Access Manager gives the shape: "just-in-time temporary privilege elevation for select principals, and to view audit logs afterwards to find out who had access to what and when." An entitlement names who may request what and for how long; a grant is time-bounded, and roles are granted "until the end of the grant duration, after which the roles are revoked by Privileged Access Manager", enforced with "time-based IAM Conditions to help ensure that access is temporary."

Four rules, each in the lab as a test, each there because of a specific failure.

Expiry rather than revocation. Nobody has to remember. The failure this prevents is the permanent break-glass credential, which is the most common version of this design and the worst — a credential that works instantly and forever, sitting in a password manager, used twice a year and never rotated.

Dual control. Without approval, one compromised account is the entire recovery plane. N5 shows the same request denied with approval required and granted without it.

No self-approval. A realistic configuration lists the same on-call engineer as both a valid requester and a valid approver, because both lists were written separately by sensible people. That is not dual control, and the evaluator refuses it explicitly rather than trusting the policy text to have got it right.

Justification required. Not for the auditor. For the responder, six weeks later, reading their own record and trying to remember what they were doing.

The property that matters most

A recovery path must not become a normal daily path. If it does, all the controls around it become friction to be routed around, and within a quarter someone has scripted a way past them.

Most approaches make it harder. The lab's approach is to make it visible, which is stronger.

Grants are bounded, so continuous use requires continuous requests — each individually justified, individually approved, and individually recorded in a log nobody can disable. T20 asserts the arithmetic: twenty-four one-hour grants do not add up to a quiet day of elevated access; they add up to twenty-four separate records with twenty-four justifications and twenty-four approvals.

This path is not blocked from routine use. It is made obvious. A blocked path gets circumvented; an obvious one gets noticed in a review.

The central authorization layer is also a shared dependency

Identity-Aware Proxy is genuinely good, and the way it is usually described is worth quoting because it contains the problem: it "lets you establish a central authorization layer for applications accessed by HTTPS, so you can use an application-level access control model instead of relying on network-level firewalls." Policies "scale across your organization. You can define access policies centrally and apply them to all of your applications and resources."

Every word of that is an argument for adopting it, and every word is also an argument for why it cannot be in your recovery path.

Central means single. Apply them to all of your applications and resources means the recovery console too, unless someone deliberately excluded it. Any layer that authenticates everything is, by construction, a dependency of everything — and a recovery path that shares an authorization layer with the path it is recovering is not independent, however good the layer is.

This is the uncomfortable shape of the whole article. The things that make a normal control plane good — centralisation, consistency, one policy applied everywhere — are precisely the things that make a recovery plane impossible. A recovery plane is deliberately inconsistent with everything else, and that is not a design flaw in it.

So: use IAP for the control plane. Then exclude the recovery path from it, explicitly, and write down that you did, because the next person to standardise access policies across the organisation will otherwise fold it back in and nobody will notice until it matters.

That advice comes with an obligation. A path carved out of your central authorization layer is a second door, and a second door is a second attack surface. It is only defensible because it has its own stronger constraints — bounded, approved, attributable, and expiring on its own — which is what the rest of this article is about. Excluding the recovery path from IAP and giving it nothing in return would be strictly worse than leaving it in.

The device is a dependency

Six of the ten questions people ask about recovery access are about credentials. Almost none are about the machine.

Laptops run a management agent that checks in with a service. Disk encryption keys may be escrowed somewhere that needs the network. The VPN client is configured by a profile that is pushed. Certificates expire and renew against an internal authority. None of that matters on a normal day, and all of it is in the dependency graph.

The specific failure to look for: a device posture check that must succeed before access is granted, evaluated by a service inside the failure domain. When it cannot be evaluated, a well-built system denies access — which is correct behaviour and exactly wrong at that moment.

Recovery devices therefore need their own row in the graph, and the honest version of that row has dependencies. A device that is genuinely independent is one that does not phone home to decide whether it is allowed to work, which is an unusual configuration and needs to be a deliberate one.

The communication path

Every recovery plane discussion is about access, and every real incident is about coordination.

Responders need to reach each other, reach the approver for a break-glass grant, and agree what is happening. In most organisations all three run through the same chat product, which authenticates through the same identity provider and resolves through the same DNS. Meta's report is explicit that this class of tooling was affected: "the total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this."

In the lab, chat is a component with dependencies rather than an assumption, and the validator flags it: restore-dns → chat → corp-dns.

Three properties make a communication path usable during the failure it is for. It must not share the failure domain, which usually means a consumer service or a phone network rather than anything the organisation runs. It must have a current contact list reachable without the systems that are down — which in practice means printed or on a device, not in the directory. And the approver for an emergency grant must be reachable through it, because dual control is worthless if the second person cannot be found.

Reachability is the most commonly missed of the three, and it is a scheduling problem rather than a technical one: an approver list that is correct and unreachable at 3am is an approver list that will be bypassed.

Testing without weakening production

Any recovery plane that has never been used does not work. This is not cynicism, it is the consistent finding of every organisation that has tried one under pressure.

But the obvious test — use the break-glass path in production — grants real privilege to real people against real systems, on a schedule, which is a standing security risk created in the name of resilience.

Separating what is being tested resolves it:

The graph can be validated continuously, at zero risk. That is what the lab does, and it needs no access to anything. Run it in CI. When someone adds a dependency edge, the validation fails, and it fails at the moment the dependency is introduced rather than during the incident. N3 is this test: one edge moved, whole plane invalid.

The human path can be exercised without the privilege. Request a grant, get it approved, reach the console, and stop. The question being tested is whether the responder can get in, whether the approver is reachable out of hours, and whether the token is where it should be. None of that requires performing a destructive action.

The destructive step belongs in an isolated environment. Never the one being protected.

What that leaves untested is the combination of real privilege and real pressure, and no honest design closes that gap. Naming it is better than claiming a drill covers it.

What the graph cannot tell you

Edges are what a graph is worth, and someone has to declare them. That is the tool's real cost and its weakest joint.

A missing dependency produces something worse than no answer: a false sense of safety, with a validator saying ok: true. This is why UNDECLARED_COMPONENT is a finding rather than a silent skip — a step that references something absent from the graph is reported, because the absence is the problem.

Dependency edges are discoverable in principle, from network policy, IAM bindings, or traces. Building that discovery is a substantially larger tool than this one and is the obvious next thing to build.

Cost, and what this is worth

Costs here are mostly not money.

Running a separate identity path is genuine duplication: another provider, another set of credentials to manage, another thing to keep current. Hardware tokens are cheap individually and a real logistics problem at scale. Printed runbooks go stale, and a stale printed runbook is more dangerous than no runbook because it will be followed.

Validation is the one thing that is nearly free. The graph is a text file, the validator runs in milliseconds with no credentials, and it can gate a pull request. Of everything in this article, that is the part to adopt first — not because it is the most important, but because it costs nothing and it tells you how bad the problem is before you spend anything on fixing it.

No one can honestly quote the cost of not having a recovery plane. Meta's report gives no restoration timeline and this article invents none. What the report does say is that engineers had to be sent physically to buildings "designed with high levels of physical and system security in mind. They're hard to get into."

Without a designed recovery plane, that is the fallback: a person in a car.

What this design does not solve

It does not find dependencies you did not declare. It is manual, and a missing edge produces false confidence.

Nor does it validate the repair itself. It also does not validate the repair: a step may be perfectly independent of the domain it repairs and still be the wrong step, or a destructive one. This article is about whether you can reach the system, not about what to do once you are in.

Nothing here addresses the data plane. Getting operators access is all a recovery plane does; restoring service is a different subject, and restoring data is Article 6's.

It cannot make emergency access safe. Elevated privilege under pressure is dangerous and remains dangerous. The controls here bound it, expire it, and record it. None of them prevents a responder with legitimate access from doing the wrong thing quickly.

And it assumes the failure domains are named correctly. Modelling "network" and "identity" as separate domains is wrong if both actually depend on one unnamed thing — a single control plane, a single region, a single vendor. The tool checks the graph you drew, not the estate you have.

A checklist

Ten questions. Any "no" is a dependency you will discover under pressure.

1. Can you name your recovery plane, as distinct from your control plane?
2. Have you drawn the dependency graph, or do you have a list of tools?
3. Do your runbooks depend on the systems they are for?
4. Does your emergency identity path federate through your primary one?
5. Does your second-factor need a network?
6. Is there anything in your recovery chain with genuinely no dependencies?
7. Does emergency privilege expire on its own, or by someone remembering?
8. Can one person request and approve their own elevation?
9. Is the record of emergency access something the user of it can suppress?
10. When did a human last complete the recovery path end to end?

Question two is the one that decides the rest. Nothing else on the list can be answered from an inventory of tools.

What must remain true

At least one tightly controlled recovery path must remain usable and auditable when the primary operating path and one major shared dependency are unavailable.

Three things hold that. A graph rather than a list, because the circular dependency is always two hops away and always invisible. Independence derived rather than declared, because a name is not an argument. And an evidence trail the person using it cannot suppress, which is the one guarantee in this article that holds against the operator rather than merely against an outsider.

If one thing transfers, draw the graph. Not the architecture diagram — the dependency graph, with failure domains, including the tools you would use to fix it. It takes an afternoon and it is usually alarming.

make verify    # 27 tests: graph, break-glass, negative controls
make report    # validate the recovery graph and regenerate the findings

make verify    # 27 tests: graph, break-glass, negative controls
make report    # validate the recovery graph and regenerate the findings

Then move one edge — put the out-of-band console behind your primary identity provider — and watch the whole plane go invalid.

Author note: this is a synthetic reference design built for this article. The estate in the lab is fictional. The lab provisions nothing and calls no Google Cloud API; it models the structure of a recovery plane, which is the part that was wrong in 2021. All findings are labelled Simulated result. No timing or recovery-duration claims appear anywhere, because no source supports one. Meta's 2021 incident report is cited as a public record of what happens when recovery tooling shares a failure domain with the outage; nothing here describes their current architecture, and no claim is made that Google Cloud would have prevented their incident. "Break glass" is industry vocabulary rather than a Google Cloud product term, and Privileged Access Manager provides one specific part of the pattern.