AI & LLMsTypeScript

Retry LLM calls with exponential backoff and jitter

LLM APIs return 429s and 5xx errors under load. This wrapper retries only the failures worth retrying, honours Retry-After and spreads retries out so many workers don't hit the API in lockstep.

#retry#backoff#resilience#llm

lib/retry.ts

type RetryOptions = {
  retries?: number;
  baseMs?: number;
  maxMs?: number;
  shouldRetry?: (err: unknown) => boolean;
  signal?: AbortSignal;
};

const RETRYABLE_STATUS = new Set([408, 409, 429, 500, 502, 503, 504]);

export function isRetryable(err: unknown): boolean {
  const status = (err as { status?: unknown } | null)?.status;
  if (typeof status === "number") return RETRYABLE_STATUS.has(status);
  // Network failures carry no status: fetch throws TypeError, Node sockets set a code
  const code = (err as { code?: unknown } | null)?.code;
  return err instanceof TypeError || code === "ECONNRESET" || code === "ETIMEDOUT";
}

function retryAfterMs(err: unknown): number | undefined {
  const headers = (err as { headers?: unknown } | null)?.headers;
  const value =
    headers instanceof Headers
      ? headers.get("retry-after")
      : (headers as Record<string, string> | undefined)?.["retry-after"];
  const seconds = Number(value);
  return value && Number.isFinite(seconds) ? seconds * 1000 : undefined;
}

const sleep = (ms: number, signal?: AbortSignal) =>
  new Promise<void>((resolve, reject) => {
    const timer = setTimeout(resolve, ms);
    signal?.addEventListener(
      "abort",
      () => {
        clearTimeout(timer);
        reject(signal.reason);
      },
      { once: true },
    );
  });

export async function withRetry<T>(
  fn: (attempt: number) => Promise<T>,
  { retries = 5, baseMs = 500, maxMs = 30_000, shouldRetry = isRetryable, signal }: RetryOptions = {},
): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn(attempt);
    } catch (err) {
      if (attempt >= retries || signal?.aborted || !shouldRetry(err)) throw err;
      // Full jitter: a random delay between 0 and the exponential ceiling
      const ceiling = Math.min(maxMs, baseMs * 2 ** attempt);
      const delay = Math.min(maxMs, retryAfterMs(err) ?? Math.random() * ceiling);
      await sleep(delay, signal);
    }
  }
}

Usage

TypeScript

import OpenAI from "openai";
import { withRetry } from "@/lib/retry";

// Disable the SDK's built-in retries so the two layers don't multiply
const client = new OpenAI({ maxRetries: 0 });

const reply = await withRetry(() =>
  client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Summarise this ticket in one line." }],
  }),
);

How it works & gotchas

01Full jitter (random() * ceiling) beats a fixed exponential delay when many workers fail at once. It spreads the retries out instead of producing synchronised waves.
02Only transient failures are retried. A 400 or 401 will fail the same way every time, so it is thrown immediately.
03Most official SDKs already retry. Pick one layer and set the other to zero, or one call can quietly turn into a dozen or more requests.
04Retrying a call with side effects (tool execution, writes) is only safe if the operation is idempotent.

Need this wired into a real system?

We build and harden production software end to end.

Talk to an engineer

More code helpers

TypeScript

Stream LLM responses from a Next.js route handler

Send tokens to the browser as the model produces them instead of making users wait for the full completion. Uses a plain ReadableStream, so it works with any client that can read a fetch body.

Python

Text chunker for RAG with overlap

Split long documents into overlapping chunks that end on paragraph or sentence boundaries, so retrieval doesn't return half-sentences. No dependencies beyond the standard library.

TypeScript

Redis sliding-window rate limiter

Limit requests per user, API key or IP across every instance of your service. A single Lua script keeps the check-and-increment atomic, so concurrent requests can't slip past the limit.