How Amazon uses timeouts, retries, and exponential backoff with jitter to handle transient failures without overwhelming struggling services.

A short note on retry budgets: cap attempts, use exponential backoff, and keep the last error for diagnostics. ### Related * Source repository * Mbin instance * Author profile on DEV

A quota 429 isn't a blip — backoff doesn't fix it, checking usage first does.

The task read simple: make the outbound call survive a temporary failure. The agent returned a retry wrapper with exponential backoff: wait 1s, 2s, 4s, cap at three attempts. The test fixture returned 500 on the first call and 200 on the third, the suite went green, and the change merged. Under real load the upstream started returning 429 Too Many Requests, and the retry made things worse instead of better. Two assumptions the agent never checked: that every failed response should be retried the same way, and that a fixed backoff schedule is enough. Both held against a generic 500 fixture and broke against the response traffic actually produced. ## A clean backoff becomes a lockstep parade Without jitter, backoff is a fixed formula: same base, same multiplier, same cap. When many clients hit the same throttled endpoint at roughly the same time, they also sleep the same computed duration and fire the next attempt at nearly the same instant. The server throttles again, they wait the same next interval, and the cluster retries in lockstep. A quiet recovery window turns into repeated synchronized spikes. Backoff shape | What happens under shared load ---|--- Fixed interval, no jitter | All clients retry together; highest overlap Exponential, deterministic | Still synchronized; overlap shrinks slowly, stays tight Exponential plus jitter | Attempts spread across the window; smoother retry curve Jitter does not reduce how often you retry; it spreads when the retries land. That distinction matters when the bottleneck is concurrency rather than a single slow request. ## 429 is not the same failure class as 500 A 500 means the server failed while handling the request; retrying later may help. A 429 means the server already accepted the request but told you to slow down, usually with a `Retry-After` header stating the earliest acceptable wait. Treating both as generic transient errors and applying one self-chosen schedule ignores the message the server sent on purpose. The agent's fixture modeled only the first shape: a server that errors and recovers. It never modeled a throttle that answers with a wait instruction, so honoring that header never entered the code. ## The rules that actually hold in traffic * Read `Retry-After` on 429 and treat it as a minimum wait, not a suggestion. Your own backoff cap should not be shorter than the delay the server already published. * Add jitter around the interval so retries fan out instead of lining up. A random offset within the computed window is enough; you do not need a custom distribution. * Split retryable classes before choosing the action. Connection failures and 503 may retry; a 401, a 403, a validation 400, or a permanent 404 should stop, never back off. * Stop the fanout under sustained throttle. After N attempts on 429, back off further or surface overload rather than keep hammering a server that asked you to wait. * Keep the idempotency question in view: backoff on a safe read is cheap; repeating a mutating call on throttle carries the same duplicate risk as any blind retry. Before baking a retry policy into code, look at what the real throttled response returns instead of inventing it. Keeping the spec and the actual request and response surface local, as Powerduck does, makes the `Retry-After` field and the response shape something you inspect directly rather than a guess. ## Write the fixture the agent skipped A retry suite that only simulates a failing-then-succeeding server keeps endorsing the lockstep path. Add a case that returns 429 with a `Retry-After` value, assert the client waits at least that long, and add a small fanout check that two simulated clients do not land their retries on the same timestamp. Resilience is not retrying every failure on a schedule you picked. It is respecting what the server told you, spreading your own retries, and stopping when the answer is permanent. Where does your retry logic currently treat 429 like a generic error, and does any test ever check the `Retry-After` header?

Resilient offline-first outbox and retry queue for Dart and Flutter with disk persistence, exponential backoff, priority scheduling, and deduplication. Changelog excerpt: - **Discovery & Metadata**: Added official funding channels and refined search discovery topics.