Md Nayeem Hossain | Software Engineer

Building Resilient Systems: Timeouts, Retries and Circuit Breakers#

Every system depends on things it does not control: payment providers, email APIs, a shared database, another team's service. Those dependencies will fail. The only choice you have is whether one failing dependency degrades a single feature or takes down your entire application.

Three patterns do most of the work: timeouts, retries and circuit breakers. Each one is simple. Getting them to work together is where the experience comes in.

Timeouts: the most important line of code#

The default timeout for many HTTP clients is "forever" or several minutes. If a dependency hangs, every request that calls it hangs too, holding a connection, memory and a worker. Under load, you run out of all three and the whole service stops responding — even for requests that never touch the slow dependency.

Set an explicit timeout on every network call:

1const res = await fetch(`${PROVIDER_URL}/payouts`, {2  method: "POST",3  body: JSON.stringify(payload),4  signal: AbortSignal.timeout(3_000),5});

Choose the value from data, not guesswork: look at the dependency's p99 latency and add some headroom. A timeout slightly above p99 cuts off pathological cases while letting normal slow requests finish.

Also think in deadlines. If the user-facing request has a two-second budget, a downstream call should not be given five seconds. Pass the remaining time down the call chain.

Retries: helpful until they are not#

Many failures are transient — a dropped connection, a brief overload. Retrying fixes them. But naive retries are dangerous:

Retrying immediately hits a struggling service again before it can recover.
Retries multiply across layers. Three layers each retrying three times turn one failure into 27 requests.
Retrying non-idempotent operations can duplicate side effects.

Rules for safe retries:

Only retry idempotent operations, or operations protected by an idempotency key.
Only retry errors that might succeed next time: timeouts, connection errors, 429, 502, 503, 504.
Use exponential backoff with jitter, so many clients do not retry in lockstep.
Cap the number of attempts and retry at one layer only, usually the one closest to the failing call.

1async function retry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {2  for (let i = 0; ; i++) {3    try {4      return await fn();5    } catch (err) {6      if (i === attempts - 1 || !isTransient(err)) throw err;7      const base = 200 * 2 ** i;8      await sleep(base / 2 + Math.random() * base); // "equal jitter"9    }10  }11}

Circuit breakers: stop calling what is broken#

When a dependency is clearly down, retries just add load and latency. A circuit breaker tracks recent failures and, past a threshold, opens: calls fail immediately without touching the dependency. After a cooldown, it lets a few test requests through (half-open). If they succeed, it closes again.

1import CircuitBreaker from "opossum";2 3const breaker = new CircuitBreaker(callPaymentProvider, {4  timeout: 3_000,5  errorThresholdPercentage: 50,6  resetTimeout: 30_000,7});8 9breaker.fallback(() => ({ status: "queued" }));10breaker.on("open", () => logger.warn({ event: "circuit.open", dependency: "payments" }));

The breaker protects the dependency, giving it room to recover, and protects you, by failing fast instead of holding resources.

Fallbacks: decide what "degraded" means#

A fast failure is only half the answer. Decide what the user sees when a dependency is unavailable:

Show cached or slightly stale data with a notice.
Queue the work for later and confirm receipt ("Your payout is scheduled").
Hide a non-essential feature, such as recommendations, while the rest of the page works.
Return a clear, honest error for features that cannot work without the dependency.

These are product decisions as much as engineering ones. Make them before the incident, not during it.

Bulkheads: contain the blast radius#

Named after ship compartments, bulkheads isolate resources so one failure cannot consume everything. Give each dependency its own connection pool or concurrency limit. If the reporting service is slow, it can exhaust its own ten connections, but checkout keeps its own pool and keeps working.

Test failure on purpose#

Resilience you have not tested is a hope, not a property. In staging, inject failures: add latency to a dependency, return errors, kill an instance. Verify that timeouts fire, breakers open, fallbacks render and alerts reach the right people. Many teams discover that their carefully configured fallback has never actually been executed.

Conclusion#

Put a timeout on every network call, retry only transient failures with backoff and jitter at a single layer, wrap fragile dependencies in circuit breakers, define fallbacks with the product team, and isolate resources with bulkheads. Then break things deliberately to prove it works. That is how one failing dependency stays a minor incident instead of a full outage.