API Error Handling That Actually Survives Production — Retries, Idempotency, and Timeouts Done Right

Most "resilient" APIs are only resilient in the happy path

Every backend has a try/catch somewhere and calls it error handling. The gap shows up the first time a downstream service gets slow, not down — slow. That's when naive retry logic turns a single struggling dependency into a full outage, because every client politely retries at the exact same interval, multiplying load on a service that was already drowning. Real resilience isn't about catching errors; it's about a small set of patterns — retries, idempotency, timeouts, circuit breaking — that have to work together, or each one individually makes things worse on its own.

Retries: the pattern most likely to cause the outage it's meant to prevent

A bare retry loop is the single most common resilience bug we see in review. The failure mode is specific and predictable:

• Retry storms. A downstream service degrades, every caller's request times out at roughly the same moment, and every caller retries at roughly the same moment — turning a partial slowdown into a synchronized load spike that finishes the job. This is why "just add retries" so often makes an incident worse, not better.
• The fix is exponential backoff with jitter, not backoff alone. Backoff without randomization still synchronizes retries across clients that all started failing at the same time — they just retry together at 2s, then together at 4s, instead of together at 1s. Jitter (a random offset added to each client's wait) is what actually breaks the synchronization.
• Retry budgets, not unlimited attempts. Cap total retries per request (2–3 is typical) and, more importantly, cap retry volume as a fraction of total traffic at the client level — if more than, say, 10% of requests are retries, that's a signal to stop retrying and start failing fast, not a signal to retry harder.
• Only retry what's actually safe to retry. A 503 or a connection timeout is usually safe. A 500 from a write endpoint is not — you don't know if the write already landed. This is where idempotency becomes load-bearing, not optional.

Idempotency: the property that makes retries safe in the first place

Retrying a GET is free — it's safe by definition. Retrying a POST that charges a card or creates an order is the actual danger, and it's the case retries exist for. The standard fix is an idempotency key: the client generates a unique key per logical operation (not per HTTP attempt) and sends it with the request; the server stores the outcome of the first request under that key and returns the same result for any repeat, instead of executing the operation again.

POST /charges
Idempotency-Key: order_8f3a2c-attempt

// Server: on first sight of this key, execute and store the result.
// On any repeat, return the stored result without re-executing.

A few details that are easy to get wrong:

• The key must be generated once per operation, client-side, before the first attempt — not regenerated on each retry, or you've defeated the entire mechanism.
• Store the key with a TTL, not forever. 24 hours is a common default — long enough to cover realistic retry windows, short enough that the idempotency table doesn't grow unbounded.
• Idempotency isn't free on the server either. It needs a lookup-then-write to happen atomically (a unique constraint on the key column is usually enough), or two near-simultaneous retries can both pass the "have I seen this key" check and both execute.
• Idempotency keys and naturally idempotent design are different tools. A PUT that sets a resource to an exact state is idempotent by design and needs no key. A POST that means "create a new charge" needs the explicit key because the operation itself isn't naturally repeatable.

Timeouts: the setting almost everyone leaves at the default

The default HTTP client timeout in most languages is either absurdly long or effectively infinite. A slow downstream dependency with no timeout doesn't fail — it hangs, and it hangs the calling thread or connection-pool slot with it. Under load, this is how one slow dependency exhausts your entire connection pool and takes down endpoints that don't even call the slow service.

• Set an explicit timeout on every outbound call, no exceptions. Not a global default buried in a client library — a deliberate value for that specific call, sized to what a reasonable response actually looks like for that dependency.
• Separate connect timeout from read timeout. A connection that won't even establish is a different failure than a connection that's open but slow to respond, and conflating them into one timeout value usually means one of the two is wrong.
• Timeout budgets should compose, not stack blindly. If your API has an overall 5s SLA and calls three downstream services sequentially, giving each a 5s timeout means a single slow dependency can already blow the whole budget. Divide the budget across the call chain deliberately.

Circuit breakers: stop calling a dependency that's already down

Retries and timeouts handle a single request's resilience. They don't stop you from hammering a dependency that's clearly unhealthy — every failed request still pays the full timeout cost before giving up, at scale, for as long as the outage lasts. A circuit breaker tracks failure rate per dependency and, past a threshold, stops sending requests entirely for a cooldown window, failing fast instead. This does two things a timeout alone can't: it protects the struggling dependency from being kept down by your own retry traffic, and it protects your own service from burning threads/connections waiting on calls that are statistically almost certain to fail.

The three states are worth knowing by name, because they map directly to the actual behavior: closed (normal — requests flow, failures are counted), open (failure threshold hit — requests fail immediately without attempting the call), half-open (after the cooldown, a small number of test requests are allowed through to check if the dependency recovered, before fully reopening the circuit).

Where these patterns actually compose

None of these work well in isolation — the value is in how they stack on a single outbound call:

1. Idempotency key generated once, client-side, for the logical operation.
2. Explicit timeout sized to the dependency, connect and read set separately.
3. Retry with exponential backoff and jitter, capped at 2–3 attempts, only for retry-safe failure classes.
4. Circuit breaker wrapping the whole call, so repeated failures stop generating retry traffic entirely once the dependency is clearly down.

Skip any one layer and the others partially compensate but leave a real gap: retries without idempotency risk duplicate side effects; timeouts without circuit breaking still waste resources on a dependency you already know is down; a circuit breaker without sane timeouts opens too late to prevent the initial damage.

The mistake we see most often in review

Not missing patterns — mismatched ones. A team adds retries to "be more resilient," ships it, and the first real outage is worse than it would have been with no retry logic at all, because the retries weren't idempotency-safe and weren't jittered. Resilience code that hasn't been tested against an actual degraded dependency (not just a hard failure) is close to untested — a dependency that's slow-but-alive is a meaningfully different failure mode than one that's cleanly down, and it's the one naive retry logic handles worst.

Building an API that needs to survive real production traffic?

We design the resilience layer alongside the API contract, not bolted on after the first incident. Tell us what you're building and we'll tell you honestly which of these patterns you actually need at your scale, and which would just be complexity you don't need yet.