Region A went, region B was healthy, and every viewer rebuffered
A one-off live event with a global audience. The feed leaves the venue over contracted transport into two cloud regions, and both regions transcode and package it — the team built the second region precisely so that losing the first would not end the event.
The first region was lost. The second was healthy the whole time and kept publishing. Every player in the world rebuffered anyway, and the ones that recovered came back to a different point in the programme.
- R1Losing an entire region of the compute provider, mid-event, must not interrupt anybody's playback. Not for the seconds it takes to notice, and not for the seconds it takes to act on having noticed.
- R2The two processing paths must not depend on each other. A protocol between them for agreeing anything is itself a component they both stop for, and is not allowed. They must nevertheless produce output a player can switch between mid-stream, without the player knowing a switch happened.
- R3The store the delivery tier reads from must not answer "not yet" for a segment that is about to exist. A live player that gets a miss retries, every player is asking for the same segment at the same moment, and the whole audience retrying at once is the outage.
Outside every boundary: The audience (all at once), Region A pipeline (transcode, package), Venue feed (one live event), Delivery tier (serves the event), Contracted transport (two paths in), an empty slot for the what fixes the segment edges, tier 1, an empty slot for the how region B is run, tier 1, an empty slot for the what delivery reads from, tier 2 Connections: Venue feed calls what fixes the segment edges — leaves the venue what fixes the segment edges calls Contracted transport — the feed Contracted transport calls Region A pipeline — into region A Contracted transport calls how region B is run — into region B Region A pipeline calls what delivery reads from — publishes how region B is run calls what delivery reads from — publishes what delivery reads from calls Delivery tier — reads Delivery tier calls The audience — the event