Failure Literacy: The Reliability Principle Stripe Learned at $1 Trillion

If Your System Has Been Running Clean for 18 Months, You Should Be Worried

High uptime only proves your system hasn’t failed recently. True reliability is how fast it recovers when it does.

Your system SLA is 99.9%, and nothing major has broken. Good news, right?

Stripe’s engineers would say that’s probably the most dangerous signal you can get.

Here’s what actually happens: a team goes 18 months without a major incident. Runbooks quietly expire. Failure drills get cancelled — nobody makes a fuss, they just stop happening. Then, at 2 a.m., an upstream service crashes. The on-call engineer opens the incident playbook and finds instructions for an architecture that was refactored two years ago.

Those two years of clean uptime didn’t prevent the failure. They made the recovery harder.

Stripe calls this the Prevention Fallacy.

The Prevention Fallacy: Uptime is a Vanity Metric

Traditional reliability engineering has a stubbornly baked-in belief: the less often a system fails, the more reliable it is.

The logic looks solid. But it can’t answer the questions that actually matter when things go wrong:

If your primary database goes offline within the next hour, who leads the recovery? Have they practiced?
How many failures in your system are currently undetected rather than prevented?
When was your runbook last updated?

In distributed systems, failure isn’t a question of if — it’s a question of when and how. At Stripe’s scale (over $1.9 trillion in annual transactions, roughly 1.6% of global GDP), every failed payment is a real financial loss.

So the real question isn’t “does it fail?” It’s: can your system recover predictably and automatically when it does?

What is Failure Literacy?

Failure Literacy is the core reliability principle Stripe has been building toward in practice: treating system failure as an expected, recoverable event rather than a catastrophic exception — at both the technical and cultural level.

The goal, distilled:

“Make failure boring before it becomes catastrophic.”

Not to eliminate failure. Make it boring. When recovery is a practiced skill, a real incident triggers an automated sequence that’s been run dozens of times — not a scramble. There’s nothing dramatic about it, which is exactly what you want at 2 a.m.

Stripe built three pillars around this.

Pillar 1: Practice Your Worst Days Every Day

Stripe’s engineering culture has an unwritten rule: don’t run a system in production that has never been broken.

It sounds paranoid. The logic behind it isn’t.

Step one is brute-force load testing. Hit the system with synthetic traffic at 20x the worst-case scenario — not to pass a test, but to find the weakest link and catch symptoms of resource exhaustion before users ever feel them.

Step two is targeted fault injection. Engineers actively break things: inject network latency, drop packets, shut down servers, pull disks, take services completely offline. They watch how the system crashes, not just whether it crashes.

Step three is the chaos testing engine. Reaching 99.9995% (five-and-a-half nines) requires resilience against multiple concurrent failures, not just single ones. Stripe’s chaos engine triggers complex compound faults in production — reproducing known failure modes and, sometimes, discovering entirely new ones.

The side effect of all this: your worst day arrives on a Tuesday afternoon during business hours, with engineers watching, instead of at 2 a.m. when nobody’s ready.

Pillar 2: Never Send a Human to Do a Machine’s Job

Stripe’s availability target is 99.9995%. That translates to 13 seconds of allowed downtime per month.

Thirteen seconds.

Humans respond to incidents in minutes. Machines respond in milliseconds. In a 13-second budget, there’s simply no room for human response — by the time anyone opens a Slack channel, the window has closed. Machines must complete detection, isolation, and recovery before users notice anything.

That constraint is why this principle exists.

13-Second Budget: Human vs Machine Response

Stripe runs over 1 million CPUs dedicated solely to monitoring, with production systems emitting millions of health indicators every second. This isn’t just for alerting — it drives automated defense. Machines handle detection, isolation, and recovery. Engineers interpret end-to-end user experience and design for long-term resilience.

Stripe also expanded what it measures. Instead of tracking only API response rates, they monitor what they call EQ (Experience Quality) metrics — the actual experience as perceived by the user, regardless of the integration method.

“Machines handle measurement and response. Humans provide the heart — interpreting the end-to-end user experience and designing the defense systems of the future.”

Shopify CTO Farhan Thawar described a good example of this at a joint session with Stripe. Taylor Swift drops some merch, millions of users hit the storefront simultaneously, and the system can’t auto-scale fast enough. Shopify’s response: Fail Open. The checkout queue tells users “You’re 30 seconds away from Taylor’s merch” instead of showing an error page.

One bulb goes out. The lamp still works. That’s the design goal.

Pillar 3: Extreme Ownership

No technical system survives a culture that doesn’t take it seriously.

Stripe runs a mechanism called “Spin the Wheel” — a weekly executive ops review in which a senior leader is cold-called at random to give a detailed postmortem of every incident under their watch: user impact, root cause, and remediation.

Nobody likes being cold-called. But the point is the forcing function: executives can’t treat reliability as something that happens below them. They have to know the operational details of their systems.

“Reliability is not owned by a committee. It is a pride that every engineer and leader puts into the systems.”

Incident reviews at Stripe aren’t run by a central SRE team — they’re run by the engineering team that owns the service. That keeps solutions focused on the full user experience, not just the team’s slice of the stack.

If the same root cause recurs, leadership is held accountable and expected to explain why it occurred. There are also carrots: leaders who reduce incidents and improve user experience get rewarded.

Two Systems Built for Trillion-dollar Scale

The cultural practices above are supported by two custom infrastructure components.

DocDB: zero-downtime elastic database layer

DocDB is Stripe’s custom database service built on MongoDB Community. It handles over 5 million queries per second across petabytes of financial data, distributed across 2,000+ shards.

Its core capability is client-transparent migration. In 2023, DocDB migrated petabyte-scale data while maintaining 99.999% uptime. Traffic cutover is handled by a fleet of Go-based proxies, completing in milliseconds to 2 seconds — imperceptible to users.

Engineers also found that sorting data by B-tree index attributes before bulk import boosted write throughput by 10x — one of those optimizations that sounds obvious in retrospect.

The Global Ledger: double-entry accounting at scale

Stripe’s ledger is grounded in double-entry bookkeeping — every money movement is recorded as both a debit and a credit, providing a mathematical guarantee of zero leakage.

Five billion ledger events daily. 99.99% of dollar volume ingested and verified within four days. 99.9999% of all money movement is fully explainable — even as business volume grew 10x.

Does any of This Apply if you’re not Stripe?

Your system probably handles a few thousand transactions a day. You don’t need a chaos engineering team. But the underlying question is the same at any scale.

Before you decide that your reliability posture is fine, ask:

When did a core service last fail in production? How long did recovery take?
How many failures in your stack are currently just undetected rather than prevented?
What percentage of your incidents do users report before your monitoring catches them?
If your primary database went offline in the next hour, who leads recovery — and have they practiced it?

If those questions are uncomfortable, good. That’s where the work starts.

Summary

What sticks with me about Stripe’s approach isn’t the chaos-engineering infrastructure. It’s a simpler reframe:

“High uptime proves a system hasn’t failed recently. True reliability is how predictably it recovers when it does. They measure completely different things.”

The 99.9999% availability number isn’t built on avoiding failure. It’s built on running failures, so often they stop being scary.

I keep thinking about that engineering team at 2 a.m., opening a playbook for an architecture that no longer exists. Two years of clean uptime led them there. The teams at Stripe have a different 2 a.m. — one where the system already knows what to do.