Post cover image

November 2, 2026

A Backup Is Not a Recovery Plan

Operator-error and ransomware-resistant disaster recovery on Google Cloud

By Abhishek Sharma

14 min read

On 31 January 2017, a GitLab engineer was trying to fix replication. Their postmortem describes what happened next with unusual directness: "an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead."

Roughly 300 GB, gone. Then they went to the backups.

Five protection mechanisms existed. The one that should have saved them had been running on a schedule, exiting cleanly, for months:

"The backup procedure was using pg_dump 9.2, while our database is running PostgreSQL 9.6."

One version mismatch. The job ran. It exited. It wrote a file. The file was unusable, and nobody knew, because the failure notification had nowhere to go: "Notifications were sent upon failure, but because of the Emails being rejected there was no indication of failure."

They lost about six hours — "roughly 5,000 projects, 5,000 comments and 700 new user accounts."

And then the sentence that makes this an engineering problem rather than an accident:

"Why was the backup procedure not tested on a regular basis? — Because there was no ownership, as a result nobody was responsible for testing this procedure."

A completed backup job is not evidence of recoverability. It is evidence that a process exited zero. Those are different claims, and the distance between them is measured in lost accounts.

Five words that are not the same thing

Most recovery conversations fail in the first minute because five distinct capabilities share a vocabulary.

Availability keeps serving when a machine, zone or region fails. It is about uptime, and it is what most architecture diagrams are actually about.

Replication keeps a second copy current. It propagates writes — including the write that deletes everything. A replica is a faithful reproduction of your mistake, applied promptly.

Backup is a point-in-time copy you can return to. Its defining property is not that it exists elsewhere but that it exists earlier.

Archive is long-term retention for compliance or history. Cheap, slow, and usually not part of a recovery path anyone has rehearsed. It earns a place on this list because it is the term most often mistaken for a backup on a cost dashboard: both are storage, both grow, and only one of them is something you have ever restored from.

Disaster recovery is the ability to restore useful business service. Not the data — the service. It includes the data, the dependencies, the order, the validation, and someone who knows how.

Replication is the confusion that costs the most. It defends against infrastructure failure and offers nothing against operator error, credential compromise, or ransomware, because in all three the adversary holds valid credentials and the system does exactly what it is told.

RPO and RTO belong to the business

Two numbers get quoted from product pages and both are usually wrong.

Recovery point objective is how much data you can afford to lose, expressed as time. Recovery time objective is how long you can afford to be down.

The product version of RPO is we back up every hour, so RPO is one hour. The business version asks a different question: how long can the corruption go undetected?

Those are two separate losses and they are worth separating. Rolling back to the last clean point costs you the data written between that point and the corruption, which is bounded by your backup interval — an hour, in this example. It also costs you everything good that was written after the corruption, because rolling back discards that too. A corruption caught in ten minutes costs an hour of granularity. The same corruption caught after three days costs an hour of granularity plus three days of legitimate work.

Backup frequency sets the first number. Detection latency sets the second, and the second is usually much larger.

That gap is the reason the lab measures RPO as the distance between the restored artifact's capture time and the source's head, rather than reading it off a schedule.

Its product version is the restore operation's duration. The business version includes deciding an incident has occurred, finding someone with authority, assembling dependencies in order, validating the result, and someone accepting that the business can reopen. The restore is often the fastest part.

An RTO that has never been executed is not an RTO. It is an estimate with a number on it.

What survives an operator

Here is the design problem stated precisely. If the adversary has valid credentials — whether malicious or simply mistaken — then any protection that identity can undo is not protection.

Stated that way, it rules out most of the obvious answers — and one case catches people on Google Cloud.

A Spanner backup has two documented properties that must be read together. First: "When you create a backup, the backup resides in the same instance, region, and project as its source database." Second: "A backup continues to exist even after its source database has been deleted, but it can't outlive its parent instance."

So the backup does survive the common accident, which is dropping a database. It does not survive deleting the instance. By default the copy and the thing it protects share a blast radius, and the documentation states the remedy in the next breath: "you can copy the backup to an instance in a separate region or project."

That copy is a deliberate act somebody has to perform. It is exactly the kind of step that is absent right up until the incident.

Cloud SQL takes a different approach that is worth noticing: point-in-time recovery "always creates a new instance; you cannot perform a PITR to an existing instance." Isolation by construction. A failed restore cannot make the incident worse, because it never touches the original.

None

The one control that holds

Bucket Lock is the mechanism in this article that genuinely defends against the credential holder, and the wording is unusually strong.

A retention policy means "objects in the bucket can only be deleted or replaced once their age is greater than the retention period." Locking it means "you prevent the policy from ever being removed or the retention period from ever being reduced." The documentation calls the action irreversible, and states plainly: "Once a retention policy is locked, you cannot remove it or reduce the retention period it has."

Deletion attempts do not silently no-op. They "fail with a 403 - retentionPolicyNotMet error." And the escape hatch is closed too: "you cannot delete a bucket with a locked policy unless every object in the bucket has met the retention period."

Note the asymmetry, because it is the design rather than a limitation: you can still increase the retention period. Protection is a ratchet. The safe operation is always available and the unsafe one never is.

Measured in the lab, with a fully privileged identity:

Action                      Result
--------------------------  -------------------------------------------
Delete the locked artifact  RETENTION_POLICY_NOT_MET
Shorten its retention       refused — a locked policy cannot be reduced
Extend its retention        permitted

Simulated result.

Row two matters more than it looks. The subtle attack is not deletion — it is to shorten retention to zero and then delete legitimately. A locked policy refuses the first step, which is why locking rather than merely setting a policy is the control.

This is the same shape as the audit-log guarantee in Article 5: most controls defend against an outsider, and a small number defend against the person holding the credentials. For operator error and ransomware, only the second kind counts.

Who is allowed to hold the boundary

Saying the production identity cannot cross the boundary raises the obvious question: who can?

It has to be a different identity, in a different project, with a different lifecycle — and that is where most implementations quietly fail. A protected copy created by the same automation that runs production, using a service account that production can impersonate, is inside the blast radius no matter which project the bytes land in. The boundary is drawn by who holds the credential, not by where the bucket is.

Three separations are worth making explicit, because each fails differently:

Who can write the protected copy. This one must be automated, frequent and unremarkable. It is the least dangerous permission on the list, and it is fine for production-adjacent automation to hold it.

Who can configure retention. Locking a policy is irreversible, so this permission destroys money if misused and protects everything if used correctly. It belongs with whoever owns the retention decision, and locking should be a deliberate, reviewed act rather than a Terraform default nobody read.

Who can restore. Recovery credentials are used rarely, under pressure, by someone who may not use them again for a year. That is the profile that argues for time-bounded elevation with an approver rather than a standing role — which is Article 5's subject and is not re-derived here.

Here is the cheap test: can the identity that runs your nightly backup job also delete last month's copies? If yes, you have one identity with two jobs and the boundary is decorative.

The dependency nobody puts in the plan

An encryption key is a recovery dependency with exactly the same standing as the backup file.

Cloud KMS is blunt about it: "After a key is destroyed, data that was encrypted with the key version can't be decrypted." And: "Destroying a key version is a permanent operation that can cause unrecoverable data loss."

There is mercy in the design — "the default scheduled for destruction duration is 30 days," and during that window you can cancel — but after it, "destruction isn't reversible."

An immutable, cross-region copy of ciphertext plus a destroyed key is not a backup. It is archaeology.

Which is why the recovery manifest treats the key as an ordered step rather than an assumption:

project → key ──┐
project → storage ──┼→ data → reference → validate
project → schema ──┘

Order is the contract. Data restored before its key is ciphertext. Before its schema, it is corruption. The lab refuses cycles, and treats a step that requires something the manifest does not declare as a blocking finding rather than a silent skip — because the undeclared dependency is the one that surfaces at 3am.

Negative control N5 shows why the key belongs in the manifest explicitly. With the key step present and the key missing, the restore stops after one step and says the payload is ciphertext. With the key step removed from the manifest, the same missing key is never checked, the restore runs further, and reports success on data it never actually decrypted. It got further and learned less.

What the lab measures

This companion module creates no cloud resources. No bucket, no lock, no KMS key, no Spanner instance. It models the structure of a protected restore, because that structure is what was missing in 2017 and a reader can check it for free.

make verify    # 25 tests, no cloud project, no credentials

Six scenarios against a synthetic double-entry ledger:

Scenario                               Outcome                      Steps    RTO    RPO  Failed invariants
-------------------------------------  ---------------------------  -----  -----  -----  -----------------
Healthy restore                        RESTORED_AND_VALIDATED         7/7  7 min  4.0 h                  0
Backup job succeeded, output unusable  BLOCKED_UNREADABLE_ARTIFACT    4/7  4 min  4.0 h                  0
Encryption key unavailable             BLOCKED_MISSING_KEY            1/7  1 min  4.0 h                  0
Manifest omits a dependency            BLOCKED_MISSING_DEPENDENCY     0/7  0 min  4.0 h                  0
Restore completes, invariants fail     RESTORED_BUT_INVALID           7/7  7 min  4.0 h                  3
After a hostile delete attempt         RESTORED_AND_VALIDATED         7/7  7 min  4.0 h                  0

Simulated result. RTO and RPO are measured inside the model and are not vendor figures or targets.

Rows two and five are the article.

Row two is the 2017 failure: a backup that ran and produced nothing usable. The lab can represent it because takeBackup returns an artifact whose payload is null while reporting success — the model must be able to express a job that worked and produced nothing, or it cannot study the failure. The restore refuses it and names the mismatch: captured by tool version 1 against schema version 2.

Row five is the one that unsettles people. Seven steps of seven completed. The restore worked. Three invariants failed.

Completion is not correctness

Negative control N4 isolates that last point, and it is the test I would show first.

Two restores. One from a clean artifact, one from an artifact missing a single credit posting. Structurally identical: the same seven steps, in the same order, deepEqual on the completed-step list passes. The only thing that separates them is the validation contract.

Steps completed  Outcome
-------------------  ---------------  ----------------------
Clean artifact                7 of 7  RESTORED_AND_VALIDATED
One posting dropped           7 of 7  RESTORED_BUT_INVALID

Simulated result.

Counting rows would have noticed one fewer and concluded the dataset was slightly smaller. The invariants notice that the ledger no longer balances, that one transaction has a single leg, and that the sum of all postings is not zero.

This is why the validation contract asserts business rules rather than schema facts. Five invariants for one ledger is the right order of magnitude — a hundred would rot, and a rotted check that still reports green is worse than no check at all.

It is also the gate most likely to be skipped, for the same reason Article 4's business-verification gate is dropped first: it is the only one that requires somebody to write down what the system is supposed to do.

None

Evidence, and why refusals count

Every path in the lab produces an evidence bundle — the outcome, the artifact, the reasons, the steps completed, the invariant results, the measured RTO and RPO, and a timestamp. Including the refusals. Especially the refusals.

A restore that blocked and left no record teaches nobody anything, and the pressure to skip the check grows every time it fires without explanation. A refusal that names the tool version and schema version is a bug report. A refusal that says "failed" is an obstacle.

The other reason evidence matters: it is the only artifact that survives the exercise. Nobody remembers in six months whether the quarterly restore test passed. The bundle either exists with a timestamp or the test did not happen.

Exercises, and finding failure during one

Any recovery plan that has never been executed does not work. This is not cynicism; it is the consistent experience of everyone who has tried one under pressure, and GitLab's own answer to "why was this not tested" was about ownership rather than tooling.

Exercises come in three sizes, at very different costs.

The manifest can be validated continuously, at zero risk. It is a JSON file. Ordering, cycles and undeclared dependencies are checkable in CI, and they break at the moment someone adds a dependency rather than during the incident.

The restore can be executed into an isolated target. Cloud SQL's PITR shape makes this the default rather than a discipline — a new instance every time, so a failed restore cannot worsen the original. The thing being tested here is whether the steps work and how long they take, which is the only honest source of an RTO.

The validation contract can be run against the restored copy, which is where you find out whether the data is usable rather than merely present.

One word on the environment itself. "Isolated" describes where the restore runs; a clean room is a stronger claim about what is in it. A restore target that shares a network with production, or pulls container images from the registry the attacker reached, or authenticates against the compromised identity provider, is isolated in name only. For operator error, ordinary project isolation is enough. For ransomware, the question is whether anything in the restore environment was reachable by the thing you are recovering from — and answering it honestly usually shortens the list of what can be reused.

What all of that costs is time and a temporary environment. What it does not cost is risk to production, provided cleanup deletes what the restore created and nothing else. Negative control N6 asserts exactly that boundary: cleanup removes the temporary copy and is refused when pointed at the protected artifact.

And when an exercise fails — which the first few will — that is the exercise working. A failed restore in a drill is a finding. The same failure discovered during an incident is an outage.

Cost, and the honest trade

Costs here are real and mostly not the storage bill.

Backup frequency buys RPO granularity linearly and costs storage linearly. It is the easiest dial and rarely the binding constraint.

Retention is where locked policies bite. Locked retention cannot be shortened, including by you, including by mistake. Storage you committed to keeping for a year is storage you are keeping for a year. That is the intended property and it means the retention decision must be right the first time.

Cross-region copies are the mitigation for the shared-blast-radius problem, and they cost both storage and egress. They are also the difference between a backup that survives losing a project and one that does not.

Test cadence is the cost nobody budgets. A quarterly restore exercise consumes a temporary environment and, more expensively, a few engineers for a day. It is also the only line item that converts an estimated RTO into a measured one.

Spanner backups "can be retained for up to a year from the time they are created," which is a ceiling worth knowing before designing a multi-year retention scheme around them.

Residency, retention, and deletion conflict

Three obligations regularly contradict each other, and this article names the conflict without resolving it.

Data residency may require copies to stay within a jurisdiction, which constrains where a cross-region copy can go — and may mean the only compliant second location shares more failure domain than you would like.

Legal retention may require keeping records for years. Deletion obligations — a right-to-erasure request, for example — may require removing a specific person's data on request. A locked retention policy makes the first easy and the second impossible for the duration.

That is a genuine conflict with legal consequences, and the engineering answer is to surface it early rather than discover it when the first request arrives. This is not legal advice and this article is not qualified to give any. What it can say is that the conflict is structural rather than hypothetical, and that "we locked retention for seven years" is a decision with a legal dimension that belongs in front of counsel before it is executed.

What this design does not solve

It does not detect corruption. Every RPO argument here assumes you notice. A corruption that goes unnoticed past your retention window is unrecoverable regardless of how good the mechanism is, and detection is a different subject.

Nor does it validate the repair. A restore can be structurally perfect and still be the wrong point in time to restore to. Choosing the recovery point is a judgement with business consequences, and nothing here makes it.

Replay is out of scope too. A restored system that re-processes events it already processed can duplicate effects — that is an idempotency problem and Article 1's subject.

The manifest is hand-written, and a dependency nobody declared is a dependency nobody restores. That is the same weakness Article 5 has, named for the same reason: the tool checks the graph you drew.

And it cannot make an untested plan trustworthy. Everything above describes what to build. Whether it works is a question only an exercise answers, and the honest position is that a plan with no execution history has an unknown RTO no matter how carefully it was designed.

Backup and DR Service is absent from this article. It is a relevant product and its documentation was not read during this work, so it is not described rather than guessed at.

A checklist

Ten questions. Any "no" is a recovery you have not got.

  1. Do you know your RPO as a business number, including detection latency?
  2. Has your RTO ever been executed, or only estimated?
  3. Can the identity that runs production delete the backups?
  4. Can it shorten their retention?
  5. Do your backups share a project, instance or region with their source?
  6. Is the encryption key in your recovery plan as an ordered step?
  7. Does your restore check the artifact's provenance, or the job's exit code?
  8. Does validation assert business invariants, or row counts?
  9. Does a blocked restore produce a record saying why?
  10. Who owns testing the restore — by name?

Question ten is GitLab's answer to their own postmortem, and it is the only one on this list that cannot be solved with architecture.

What must remain true

Recovery is complete only when protected data can be restored into a trusted environment, dependencies can be re-established in the correct order, application invariants pass, and the result is supported by auditable evidence.

Three things hold that. A copy the production identity cannot destroy, because operator error and ransomware both arrive holding valid credentials. A restore that validates the artifact rather than the exit code, because a job that succeeded and produced nothing is the documented failure mode. And a validation contract about the business rather than the schema, because seven steps of seven can complete over a ledger that does not balance.

If one thing transfers, go and try to delete yesterday's backup with your production credentials. Not in a meeting — actually attempt it. If it works, you have replication with a longer name.

make verify    # 25 tests: protection, restore, negative controls
make report    # regenerate every figure quoted above

Then change one thing — set retentionLocked to false — and watch which row moves.

Author note: this is a synthetic reference design built for this article. The dataset, the ledger and the estate are fictional. The lab creates no cloud resources and calls no Google Cloud API; it models the structure of a protected restore, which is the part that was missing in 2017. All figures are labelled Simulated result, and the RTO and RPO figures are measured inside the model rather than quoted from any vendor page — presenting an untested RTO as achieved is the thing this article argues against. GitLab's 2017 postmortem is cited as a public record of the difference between a completed backup job and proven recoverability; nothing here describes their current architecture, and no claim is made that Google Cloud would have prevented their incident. The section on residency, retention and deletion names a real legal conflict and deliberately does not resolve it; it is not legal advice.