wtf( )unctionsystem design, drawn
← all problemsResilienceHard

The backups were untouchable and nobody could get anything back

The copies survived the attack exactly as designed. Production's automation could add to them and not remove them, retention could not be shortened by anybody, and every attempt to weaken either was recorded somewhere the attacker could not reach.

Then the recovery began, and it took eleven days. Nothing was missing. Everything about getting it back turned out to be a plan rather than a thing that had been done.

  1. R1Losing one region must not lose the copies. A second copy that fails when the first one does has told you nothing you did not already know about your ability to recover.
  2. R2A restore must land somewhere clean, and exercising a restore must not touch production — not to read a value, not to look something up, not to check a version. If the drill needs production, the drill is unavailable exactly when it is needed.
  3. R3The copies are encrypted. Whatever the restore needs in order to read them must be held somewhere that outlives the account under attack, and must not be something an application instance was carrying.
Compose the recovery side. Tier 1 is where a second copy lives and what a restore lands in, tier 2 what the restore needs in order to read anything.
Components — tap one, then tap a slot on the diagram
?None of these are about stopping the attacker. They are about the week after, which is the half that gets designed on a slide.

Boundaries, outermost first: THE ACCOUNT UNDER ATTACK: Production (assume it is theirs; FAILED: compromised), The copies (write-once, retained) Outside every boundary: an empty slot for the where the second copy lives, tier 1, an empty slot for the where a restore lands, and is rehearsed, tier 1, an empty slot for the what makes the copies readable, tier 2 Connections: Production sends outbound traffic to The copies — adds, never removes The copies publishes to where the second copy lives — copied onward where the second copy lives calls where a restore lands, and is rehearsed — restored into what makes the copies readable controls where a restore lands, and is rehearsed — unlocks

Productionassume it is theirscompromised
The copieswrite-once, retained