Reliability: Timeouts, Retries, Circuit Breakers and Idempotency

Prowid Photos has a notifications service: it counts the number on the «bell» in the site header. On Friday evening it starts answering in 5 seconds instead of 20 ms — and the feed, which needed just one number, stops opening for everybody, although the photos and the database are fine. In this lesson we will see how a small failure turns into an outage and which techniques stop it: timeouts, retries, idempotency, circuit breakers, rate limiting, degradation and health checks.

Everything breaks: cascading failure

The network drops packets, a deploy restarts a server, a disk fills up. In a system made of several parts, something is broken almost all the time (availabilities multiply — remember the first lesson). Reliability does not mean «nothing breaks»; it means «when a part breaks, the rest keeps working».

What does a hung service do? The number of requests «hanging» in the system at the same moment equals requests per second × response time. At 350 RPS and a 50 ms answer that is 350 × 0.05 ≈ 18 requests. If each of them waits 5 seconds for notifications, it is 350 × 5 = 1750. Each one holds a socket, memory and maybe a database connection. Instances hit their limits, the balancer sees failing health checks, and users press «refresh» and add more load. This is a cascading failure: the outage of a secondary part drags the main parts down with it. Our goal is to keep a failure where it happened.

Timeouts: every call has a limit

You call a friend and nobody picks up. After half a minute you hang up instead of listening to the ringing until evening. Code must do the same. By default fetch in Node.js can wait for minutes, and many drivers have no limit at all. The modern way is AbortSignal.timeout(): when the time is up, fetch fails with a TimeoutError.

export async function fetchJson(url, { timeoutMs = 2000, ...init } = {}) {
  const res = await fetch(url, {
    ...init,
    signal: AbortSignal.timeout(timeoutMs), // abort after timeoutMs
  });
  if (!res.ok) {
    const err = new Error(`HTTP ${res.status}`);
    throw Object.assign(err, { status: res.status });
  }
  return res.json(); // the same signal also covers reading the body
}

try {
  await fetchJson('http://notifications:4001/unread/42', { timeoutMs: 300 });
} catch (err) {
  if (err.name === 'TimeoutError') console.warn('notifications is too slow');
  else throw err;
}

How do you choose the value? Open the percentiles from the first lesson: if the p99 of notifications is 200 ms, set 300–500 ms, not 30 seconds «just in case». A chain of calls needs a time budget: the feed answers in 300 ms at p95, so three sequential calls of 2 seconds each are impossible.

⚠️ A timeout does not cancel the work on the server: notifications may have executed the request, the answer just did not arrive in time. A retry after a timeout can do the job twice — more about this below.

Retries: once more, but wisely

Many failures are short: the network blinked, a server restarted during a deploy. A retry rescues such cases, but not everything should be retried: