practiceEngineering & architecture

Retry

A controlled repeat attempt for an operation that failed in a way likely to be temporary.

Download Markdown (.md)

01What it means

A retry is another attempt at an operation after a failure or missing response. A retry policy decides which failures are transient, how many attempts are allowed, how long to wait, and when to stop. Safe policies combine bounded attempts, backoff, randomness, deadlines, and idempotent operations.[1][2]

Bounded recovery policy Retry only transient failures within one deadline and attempt budget
  1. 01AttemptOperation under deadline
  2. 02Classify failureTransient, permanent, ambiguous
  3. 03Check safetyIdempotent or stable key
  4. 04Back off + jitterHonor server guidance
  5. 05Retry or stopBudget, deadline, final outcome

Backoff and jitter reduce synchronized load; idempotency prevents repeated effects after ambiguous outcomes.

02Why it matters

Brief network faults, contention, and service interruptions are normal in distributed products. Selective retries can hide short disruptions, but uncontrolled retries multiply load, increase latency and cost, and can prevent a struggling dependency from recovering.

03When to use it

Retry when a documented signal indicates a temporary condition and the operation is idempotent or otherwise protected from duplicate effects. Fail promptly for invalid input, permission errors, permanent business rejection, expired user intent, or when the overall deadline cannot accommodate another attempt.

04How it works

  1. Classify errors into retryable, non-retryable, and ambiguous outcomes.
  2. Confirm the operation is idempotent or attach a stable idempotency key.
  3. Set an overall deadline and a small maximum number of attempts.
  4. Wait using exponential backoff with jitter, honoring server retry guidance where available.
  5. Stop when the deadline, attempt budget, or non-retryable condition is reached.
  6. Measure added traffic, final success, latency, and correlated retry storms.

05A practical example

Example

A mobile client times out while refreshing a product catalog. It retries twice within the screen's deadline using randomized backoff. It does not retry an invalid authentication token and stops when the API returns a retry delay beyond the remaining deadline.

06Where it breaks

Retries cannot repair persistent faults or insufficient capacity. They add load and tail latency, can repeat side effects, and may synchronize into a retry storm. Multiple layers retrying independently can multiply attempts far beyond the intended limit.

Common mistakes

  • Retrying every error, including invalid requests and authorization failures.
  • Using immediate or unbounded retries without backoff, jitter, or a deadline.
  • Retrying a non-idempotent operation without duplicate-effect protection.
  • Configuring retries at several layers without one end-to-end attempt budget.

07Sources

  1. Timeouts, Retries, and Backoff with JitterAmazon Web Services · Institutional engineering guidanceTimeouts, Retries, and Backoff with Jitter
  2. Retry PatternMicrosoft Learn · Institutional architecture guideRetry Pattern
View saved topics

Start typing to search titles, definitions, categories, and frameworks.