The shape every outage takes
Large outages look different in the news and alike in the post-incident reports. Nearly all of them follow the same path:
- A trigger: a change, a race, a malformed file.
- A fault: one component misbehaves.
- Propagation through a shared dependency: DNS, identity, configuration, a control plane, time.
- Amplification: retries, thundering herds, caches expiring together, crash loops.
- The outage.
- Recovery, slowed by tooling that shares fate with what failed.
How a small fault becomes a large outage
1. Trigger
A change, a race, a malformed file
2. Fault
One component misbehaves
3. Shared plane
DNS, identity, configuration, a control plane, time
4. Amplification
Retries, herds, caches expiring together, crash loops
5. Outage
Many services, many regions
6. Slow recovery
The tools to fix it share its fate
Two steps on that path are where design makes the biggest difference. Propagation is decided by what components share, which you can map in advance. Amplification is decided by retry and backoff behaviour, which you can set in advance. The rest of this page is about both.
Five incidents, read closely
These five were chosen because each provider published enough to see the mechanism. The dates and times are in each provider's own time zone.
AWS us-east-1, 19 and 20 October 2025
What happened. A latent race condition in DynamoDB's DNS automation, between the components AWS calls the Planner and the Enactor, left an empty DNS record for dynamodb.us-east-1.amazonaws.com. DynamoDB API errors ran from 11:48 PM to 2:40 AM PDT. Console sign-ins using an IAM user saw increased authentication failures from 11:51 PM to 1:25 AM. Dependent services failed, EC2 launches were impaired and throttled, and the event ended at 2:20 PM PDT on 20 October after three distinct periods of impact.
A detail worth getting right. Customers using DynamoDB global tables could still connect to their replica tables in other Regions; what they saw was prolonged replication lag. Earlier summaries said "IAM updates and global tables were affected", which is broader than the post-event summary supports.
What it teaches. The multi-region data plane worked. The regional control and identity paths didn't. Local credentials survived, which is the pattern called fail-static identity. Version-fence your DNS plan updates, and make sure recovery paths don't depend on the service that failed. AWS disabled the automation worldwide.
Google Cloud, 12 June 2025
What happened. A new quota-checking code path in Service Control, added on 29 May, lacked error handling and wasn't behind a feature flag. At about 10:45 AM PDT a policy with blank fields was inserted, and its metadata "replicated globally within seconds". The blank fields hit the unprotected code, causing a null pointer and a crash loop in every region, from 10:51 to 18:18 PDT. As tasks restarted in us-central1, they overloaded the Spanner database underneath without randomised backoff, and that region took up to about 2 hours 40 minutes to recover fully.
What it teaches. Google's report notes that Service Control is a regional service with a regional datastore, yet the policy data replicated globally. That mismatch has a name in this atlas: regional compute, global state. It made a regional service a global failure domain. Google's remediation was to modularise the checks so that the functionality is isolated and fails open, which is right for a quota check and would be wrong for an access check. The next section explains the difference.
Azure Front Door, 29 October 2025
What happened. An inadvertent tenant configuration change introduced an invalid or inconsistent state that caused a significant number of AFD nodes to fail to load properly, from 15:45 UTC to 00:05 UTC the next day. Microsoft 365 and the Azure portal were affected. The portal was failed away from AFD at 17:26, and configuration changes were blocked at 17:30 while a last-known-good configuration was rolled out.
The deeper mechanism. Microsoft's later engineering post explained that a different sequence of configuration changes across two control-plane versions produced incompatible metadata, and that the bad configuration passed the protection system before a delayed asynchronous processing task crashed on it. A separate AFD incident on 9 October involved the protection system being bypassed during a manual clean-up.
What it teaches. Control-plane version skew can make configuration incompatible, so configuration needs a schema version and a skew policy between producers and consumers, as Kubernetes has for its components. Validation has to exercise the real consumer, including its asynchronous stages. And management tooling must not depend on the edge it manages.
Cloudflare, 18 November 2025
What happened. A change to permissions in a ClickHouse database caused a query to return duplicate rows. The Bot Management feature file generated from that query doubled in size, past a hard limit that Cloudflare's postmortem describes as set to 200, well above its normal use of about 60 features. The proxy panicked. Failures began at 11:20 UTC. Because the file was regenerated every few minutes, the network flapped between working and failing. Core traffic was largely flowing normally by 14:30, and all systems were normal by 17:06. An unrelated failure of Cloudflare's off-platform status page at the same time suggested to responders that they were under attack.
What it teaches. Generated configuration is configuration whose producer is a query, so a permission change upstream of it is a configuration change. Validate generated configuration for size and cardinality, not just schema. And a non-critical module must not be able to crash the critical path.
CrowdStrike Channel File 291, 19 July 2024
What happened. A new IPC Template Type defined 21 input parameter fields, but the integration code supplied only 20[1]. The Content Validator checked against 21. A content update with a non-wildcard criterion in the 21st field made the sensor read past the end of the input, an out-of-bounds memory read, crashing Windows hosts between 04:09 and 05:27 UTC. Microsoft estimated 8.5 million devices, under 1% of Windows machines; later reporting said more than 8 million. Each had to be recovered by hand.
What it teaches. Content needs staged rings with bake time, just as code does, and customers need control over when it rolls out. Add runtime bounds checks, and test with the non-wildcard values production will actually use.
What the five have in common
All five were data or configuration that propagated faster than code would have been allowed to. Three of them also impaired the tools people needed to see or recover. Two (Google and Azure Front Door) involved a mismatch between where a component ran and how far its data travelled.
The five ways a check can fail
Saying "authorization never fails open" sounds safe, and then you read Google's remediation, which made its checks fail open deliberately. Both are right, because they're talking about different kinds of check. Every check needs a declared failure mode, and there are five:
| Mode | Use it for | What it requires |
|---|---|---|
| Fail-closed: deny when unsure | Access to protected data; granting privilege | A break-glass path |
| Fail-staticWhen a check can't get a fresh answer, it keeps using the last known good one, up to a declared maximum age, instead of allowing or denying everything.The five fail modes: use the last known good decision or data, up to a declared age | Authorization within a staleness budget; configuration; DNS fallback | A declared maximum age |
| Fail-open: allow when unsure | Quota, rate limits, budgets; advisory enrichment | An alarm, a time limit and bounded exposure |
| Fail-degraded: serve a reduced function | AI generation falling back to search-only; read-only mode | A defined experience and SLO |
| Fail-to-manual: a human decides | Irreversible side effects: payments, deletions | A staffed approval path |
And checks fall into four classes:
- Protective checks (authentication, and authorization over protected data or side effects) fail closed, or fail static within a declared revocation window.
- Economic and availability checks (quota, rate limit, budget) may fail open, with alarms, because over-admission is bounded and reversible.
- Enrichment checks (telemetry, advisory risk scoring) fail open.
- Safety interlocks on irreversible actions fail to manual.
A fail-open path is itself security-relevant configuration. Version it, alarm on it, and put a time limit on it.
The failure matrix
| Failure | How it spreads | How you detect it | How you prevent or contain it |
|---|---|---|---|
| Identity-provider outage | New sign-ins and token refreshes fail | Synthetic sign-ins | Design token lifetimes; break-glass outside Conditional Access |
| DNS failure | Every client; TTLs delay recovery | Resolution SLIs from several vantage points | Fenced automation; static fallbacks |
| Certificate expiry | Everything fails at the same moment | Expiry inventory and alerts | ACME or managed certificates |
| Database failure | Connection storms | Lag and error rates | Tested failover; idempotent clients |
| Storage corruption | Replicas copy it | Integrity checks | Versioning, object lock, separate backups |
| Region or provider failure | Global features homed in that region | External monitoring | Static stability; pre-provisioned capacity |
| Model API outage | Every AI feature | Per-provider SLIs | Policy-equivalent fallback |
| GPU or quota exhaustion | Queues grow; 429s cascade | Queue time, KV utilisation, headroom | Reservations, admission control, a smaller fallback model |
| Network partition | Split brain | Path monitoring | Quorum with fencing |
| Corrupted data | Pipelines, indexes and answers | Quality checks | Data contracts, versioned tables, rewinding along lineage |
| Ransomware | Lateral movement through shared credentials | EDR, canaries | A separate backup identity plane; immutability |
| Credential theft | The attacker acts as the identity | Anomalous Graph use | Phishing-resistant MFA; no long-lived keys |
| Prompt injection or a malicious tool | Tools act for the attacker | Tool-call anomaly audit | Authorization inside tools; approvals; signed, pinned MCP servers |
| Logging outage | Blind operations; a compliance gap | Pipeline health | Local buffering; an independent failure domain |
| Backup or restore failure | Silent until you need it | Restore drills | Rebuild from IaC and restore data, tested together |
| Configuration drift | Inconsistent security | Drift detection | GitOps; block out-of-band changes |
| Clock skew | Tokens rejected, leases broken, logs misordered | An NTP-offset SLI | Redundant time sources; skew tolerance; monotonic clocks for timeouts |
| Signing-key rotation | Mass 401s | JWKS fetch errors | Overlap windows; publish keys before use |
| Admission-webhook outage | Every write blocked | API error rate | Scoped failure policies; a webhook path outside the cluster |
Blast radius: a method
"What would this failure take with it?" is a question you can answer before the failure, if you map the dependencies.
- Draw the dependency graph with typed edges: depends on (at runtime), authenticates as, resolves via (DNS), keyed by (KMS or PKI), configured by, observed by, recovered by.
- For a failed node F, list its direct dependents, one edge away.
- Add its indirect dependents: the transitive closure along runtime edges.
- Add the shared-plane dependents: everything that shares F's control plane, configuration producer, identity issuer, DNS zone, telemetry pipeline or backup system.
- Subtract what is statically stable: dependents whose edge to F is only needed for change, each with a declared survival time, such as the remaining token lifetime.
- Flag recovery coupling: does restoring F need F, or anything in F's blast radius?
The blast radius is everything found in steps 2 to 4, less whatever step 5 removes. It changes with time, because cached tokens, DNS answers and certificates all expire.
| Dependency | Direct dependents | Indirect and shared | Statically stable for | Recovery coupling |
|---|---|---|---|---|
| Identity provider | Sign-ins, token refresh, workload identity exchanges | All SaaS, CI/CD, agents, admin portals | The remaining token life: about an hour by default, up to 28 hours with CAE | Break-glass must not need the identity provider or Conditional Access |
| DNS | Every call by name | Private endpoints, the identity provider, certificate checks (OCSP, CRL) | The cached TTL | Recovery tools that resolve through the same zone |
| KMS | Decryption on read; envelope keys | Databases, storage, secrets, backups | Cached data keys, depending on the provider | Deleting a key or locking its policy makes backups unreadable |
| PKI | TLS and mTLS handshakes | Service mesh, identity-provider signing, device authentication | Until certificates expire | The offline procedures for the issuing CA |
| Network transit | Cross-network and on-premises flows | Hybrid DNS, Active Directory, backups | None: it's live | An out-of-band management path |
| Database | Application reads and writes | Caches (stampedes), indexes (CDC stalls) | Read caches | Connection storms on failover |
| Object storage | Artefacts, the data lake | Image pulls, model weights, backups | Local copies | Restores need storage in another domain |
| AI gateway | Every AI feature | Agents, metering, audit | None, unless clients bypass it, which they mustn't | A last-known-good gateway configuration |
| Model provider | Generation | Agents, evaluations | Only a policy-equivalent fallback | Quota in the fallback region |
| Kubernetes control plane | Scheduling, scaling, secret updates | Operators, admission | Existing pods keep running | Webhooks that fail closed block the fix |
| CI/CD | Deployments | IaC changes, policy rollout | Running systems | The pipeline identity is Tier-0; keep a manual emergency path |
| Observability | Detection, paging | Audit evidence, SLOs | Local buffers | Independent paging and status |
| Backup | Restore | Ransomware recovery, legal hold | Not applicable: it's latent | A separate identity, keys and control plane |
The Tier-0 set
The Tier-0 set: components whose failure or compromise takes everything with it
Identity provider
And every identity that can grant permissions
DNS
Including the private zones behind private endpoints
KMS and PKI
Keys, signing certificates, issuing CAs
Edge and CDN
And the tooling that configures it
AI gateway
Every AI feature routes through it
CI/CD and IaC state
Can recreate or destroy everything
Backup control plane
The last line, so it needs its own identity
Time source
Leases, token expiry, certificates and logs all read it
A Tier-0A component whose failure or compromise takes everything else with it, such as the identity provider, DNS or the key management service.The Tier-0 set component is one whose compromise or failure compromises or fails everything else. The revised set is: the identity provider and every permission-granting identity; DNS; KMS and PKI; the edge and CDN; the AI gateway; CI/CD and IaC state; the backup control plane; and the time source that leases, tokens, certificates and logs all read. The last one is the newest addition, and the one most estates have never mapped.
Map the Tier-0 set against the failure matrix first. Most propagation paths run through it.