A typical small product talks to more services than its team can list from memory: a database, an object store, a payment provider, a transactional mail service, an authentication provider, analytics, error tracking, maybe a search index and a queue. Each is a separate company with its own bad days.
The readiness guide asks whether your app degrades gracefully or breaks, because the answer says a lot about how it was built. A product that returns a blank error page because analytics is unreachable is telling you that its failure handling was never really designed.
Dependencies are not equal, and treating them as though they are is the problem
Sort yours into three buckets. This takes ten minutes and changes how you write the code.
Critical: without this, there is no product. Your primary database, usually, and your authentication if you did not build it. When these are down you are down, and the work is to fail clearly and tell people.
Important: a major feature stops, but the rest works. A payment provider is the classic case. Customers cannot buy, which matters, but existing customers can still use what they already have. Search is often here too.
Peripheral: analytics, error reporting, feature flags, marketing pixels, the customer chat widget. None of these should be able to affect whether a page renders, and all of them regularly do.
The single highest-value thing most teams can do here is make absolutely certain nothing in the third bucket can take down a page. A synchronous call to an analytics endpoint in a request handler is a fuse wired directly from somebody else's outage to your homepage.
Slow is worse than down
This is the part that surprises people. A dependency that refuses connections fails fast, and your code moves on. A dependency that accepts the connection and then takes ninety seconds to answer holds one of your workers hostage for ninety seconds. Enough of those and every worker is waiting, and your application is down even though nothing in it is broken.
Two defences, both unglamorous:
Set explicit timeouts on every outbound call. Most HTTP clients default to no timeout or an extremely generous one, which is exactly wrong for a request a user is waiting on. A few seconds is usually right. The correct question is not "how long might this take" but "how long is this worth waiting for before I give the user an answer."
Add a circuit breaker for anything you call often. After a handful of consecutive failures, stop calling for thirty seconds and fail immediately instead. This turns a slow cascading failure into a fast, contained one, and it stops you adding load to a service that is already struggling.
Deciding what to do when it fails
For each important dependency there are four honest options. Pick one deliberately per feature rather than letting the default be an exception.
- Serve stale. Cache the last good response and use it. Right for things that change slowly: plan limits, feature flags, configuration, currency rates.
- Queue it. Accept the work, store it, do it when the service returns. Right for anything asynchronous: emails, webhooks you send, analytics events, report generation.
- Degrade the feature. Turn off the part that needs the dependency, leave the rest. Search unavailable, browsing fine.
- Refuse honestly. Tell the user it is not working right now. Correct for anything involving money or anything where a silent retry could produce a wrong outcome.
Payments deserve the last one specifically. Queueing a payment that the customer believes succeeded, or retrying one without an idempotency key, are both ways of turning an outage into a reconciliation problem that lasts weeks. An honest "we could not take payment, nothing has been charged, please try again" is a much better afternoon for everyone.
What the user sees
The technical handling is half of it. The other half is the message, and this is where most products are worst.
A generic "Something went wrong" tells the customer nothing, so they retry, which adds load, and then they email support, who also do not know. Three sentences fix it: what is not working, what still works, and what happens next.
Card payments are temporarily unavailable because of an issue at our payment provider. Everything else is working normally, and nothing has been charged. We are watching it and will update this page.
That costs one afternoon to build once and saves the support load of every future incident. It also reads as a company that knows what is happening in its own system, which is worth something on the day you most need the benefit of the doubt.
Retries, and how they make things worse
The instinct when a call fails is to try again. Done naively, this is how a brief problem at a dependency becomes a long one.
If every one of your servers retries immediately, three times, the moment a service starts failing, you have tripled the load on something that is already struggling. Everyone else's application is doing the same thing. The service cannot recover because the retries will not let it.
Three rules keep retries useful:
- Back off exponentially. Wait a second, then two, then four. Not three attempts in a hundred milliseconds.
- Add jitter. A random fraction added to each wait, so that all your servers do not retry in lockstep. Without it, synchronised clients produce a pulse of traffic every time the delay elapses.
- Only retry what is safe to repeat. Reading something twice is harmless. Creating something twice may charge a customer twice. For anything that changes state, use the idempotency key the provider offers, and if it does not offer one, treat the operation as non-retryable.
That last distinction is worth being strict about. A retry on a payment without an idempotency key is not resilience; it is a duplicate charge waiting for a bad afternoon.
Testing it
You cannot claim any of this without having tried it. In staging, for each important dependency, simulate both failure shapes:
- Refused: block the host, or point it at a port with nothing listening. Confirm the page still renders and the message is the one you wrote.
- Hanging: point it at a stub that accepts the connection and sleeps for a minute. This is the one that finds missing timeouts, and it is the one almost nobody tests.
Do the same for the peripheral bucket, where the expected result is that nothing visible happens at all. If turning off error tracking changes what a user sees, something is wired wrong.
Where to start
If you do one thing: put a timeout on every outbound call, and move everything in the peripheral bucket out of the request path. Those two changes are usually a day of work and they remove the majority of the ways someone else's outage becomes yours.
The rest, the caching, the queueing, the written messages, is worth doing in the order your dependency list suggests. You will find, as most teams do, that the list is longer than you expected and that at least one item on it is something nobody remembers adding.