Designing idempotent APIs and webhooks that survive retries

Every distributed system eventually sends the same request twice. A mobile client times out and retries. A load balancer resets a connection after the server already committed. A payment provider redelivers a webhook because your endpoint took too long to answer. None of these are edge cases; they are the normal operating conditions of software that talks over a network.

The question is not whether duplicates will arrive, but what happens when they do. If the answer is a customer gets charged twice or an order ships twice, you have a correctness bug that only shows up under load or during incidents, which is exactly when you can least afford it. The fix is to make your write operations idempotent: performing them once or many times leaves the system in the same state and returns the same result.

This article walks through the patterns we use in production backends: idempotency keys for client-facing APIs, database constraints as the final line of defence, deduplication for inbound webhooks, and the transactional outbox for reliably publishing events.

Why retries are unavoidable

Consider a client that sends POST /payments and never receives a response. From its point of view there are three possibilities: the request never reached the server, the server crashed mid-processing, or the server finished and the response was lost on the way back. The client cannot tell these apart. The only safe move is to retry, and the only way a retry is safe is if the server can recognise it.

HTTP already gives us part of the answer. GET, PUT and DELETE are defined as idempotent: putting the same resource twice yields the same state. POST and PATCH are not. In practice most business operations, such as creating an order, capturing a payment or sending an invite, map naturally to POST, so we need an explicit mechanism.

• At-least-once delivery is the default for queues, webhooks and most SDKs with retry logic.
• Exactly-once processing is something you build on top, by deduplicating at the point where state changes.
• The deduplication must happen inside the same transaction as the state change, or there is a window where duplicates slip through.

Idempotency keys for client-facing APIs

The widely used pattern is an Idempotency-Key header. The client generates a unique value (a UUID is fine) per logical operation and sends the same key on every retry of that operation. The server stores the key with the outcome and, on a repeat, returns the stored response instead of executing again.

Three details make or break the implementation. First, the key must be scoped to the caller, otherwise two tenants can collide. Second, you should store a fingerprint of the request body so that reusing a key with a different payload is rejected rather than silently returning an unrelated result. Third, you need to handle the case where the first request is still in flight when the retry arrives.

CREATE TABLE idempotency_keys (
  account_id     uuid        NOT NULL,
  key            text        NOT NULL,
  request_hash   text        NOT NULL,
  status         text        NOT NULL DEFAULT 'in_progress', -- in_progress | completed
  response_code  int,
  response_body  jsonb,
  created_at     timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, key)
);

-- Keys only need to live as long as clients might retry.
CREATE INDEX idempotency_keys_created_at_idx ON idempotency_keys (created_at);

The primary key on (account_id, key) is doing the real work. Two concurrent requests with the same key cannot both insert a row; one wins and the other gets a unique violation. We lean on that instead of trying to coordinate with application-level locks.

import { createHash } from "node:crypto";
import type { Pool } from "pg";

type Result = { status: number; body: unknown };

export async function createPaymentIdempotent(
  db: Pool,
  accountId: string,
  key: string,
  payload: { amount: number; currency: string; customerId: string },
): Promise<Result> {
  const requestHash = createHash("sha256").update(JSON.stringify(payload)).digest("hex");
  const client = await db.connect();

  try {
    await client.query("BEGIN");

    // Claim the key. ON CONFLICT DO NOTHING lets us detect a repeat without an error.
    const claimed = await client.query(
      `INSERT INTO idempotency_keys (account_id, key, request_hash)
       VALUES ($1, $2, $3)
       ON CONFLICT (account_id, key) DO NOTHING
       RETURNING key`,
      [accountId, key, requestHash],
    );

    if (claimed.rowCount === 0) {
      await client.query("ROLLBACK");
      const { rows } = await db.query(
        `SELECT request_hash, status, response_code, response_body
         FROM idempotency_keys WHERE account_id = $1 AND key = $2`,
        [accountId, key],
      );
      const existing = rows[0];
      if (existing.request_hash !== requestHash) {
        return { status: 422, body: { error: "Idempotency-Key reused with a different payload" } };
      }
      if (existing.status !== "completed") {
        return { status: 409, body: { error: "Request with this key is still in progress" } };
      }
      return { status: existing.response_code, body: existing.response_body };
    }

    const { rows } = await client.query(
      `INSERT INTO payments (account_id, customer_id, amount, currency)
       VALUES ($1, $2, $3, $4) RETURNING id, amount, currency`,
      [accountId, payload.customerId, payload.amount, payload.currency],
    );
    const response: Result = { status: 201, body: rows[0] };

    await client.query(
      `UPDATE idempotency_keys
       SET status = 'completed', response_code = $3, response_body = $4
       WHERE account_id = $1 AND key = $2`,
      [accountId, key, response.status, JSON.stringify(response.body)],
    );

    await client.query("COMMIT");
    return response;
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

Because the key claim, the business write and the stored response all commit in one transaction, a crash at any point either leaves nothing behind (so the retry runs cleanly) or leaves a completed record (so the retry gets the cached answer). A concurrent duplicate blocks on the unique index until the first transaction commits or rolls back, then sees the outcome.

Let the database enforce uniqueness

Idempotency keys protect against client retries, but they are only one layer. Many operations have a natural key that should be unique regardless of how the request arrived: one subscription per customer per plan, one invoice per order, one membership per user per workspace. Encode those rules as constraints.

-- One active subscription per customer and plan.
CREATE UNIQUE INDEX subscriptions_one_active_idx
  ON subscriptions (customer_id, plan_id)
  WHERE status = 'active';

-- Upsert instead of check-then-insert.
INSERT INTO workspace_members (workspace_id, user_id, role)
VALUES ($1, $2, 'member')
ON CONFLICT (workspace_id, user_id) DO NOTHING;

The anti-pattern to avoid is check then insert: SELECT to see whether a row exists, then INSERT if it does not. Under concurrency two requests both see nothing and both insert. A unique constraint turns that race into a deterministic error you can handle, and ON CONFLICT turns it into a no-op.

Deduplicating inbound webhooks

Webhook providers almost universally deliver at least once. They retry on timeouts, on non-2xx responses and sometimes for reasons of their own. Each event usually carries a unique identifier, and that identifier is your idempotency key.

1. Verify the signature before doing anything else, using the raw request body.
2. Insert the event id into a processed_events table with a unique constraint. If the insert conflicts, return 200 immediately: you have seen it before.
3. Persist and acknowledge quickly. Store the event and enqueue the work, then return 200. Heavy processing inside the request is what causes provider timeouts and redeliveries in the first place.
4. Process from the queue with handlers that are themselves idempotent, because the queue is at-least-once too.

Also remember that webhooks can arrive out of order. A subscription.updated event may land before subscription.created. Rather than trusting event order, treat the event as a signal and fetch the current state of the object from the provider's API, or compare a version or timestamp before applying changes.

Publishing events reliably with the outbox pattern

The mirror image of the webhook problem is publishing your own events. A common bug looks like this: commit the order to Postgres, then publish order.created to a broker. If the process dies between the two, the order exists but nobody downstream hears about it. Reversing the order is worse: consumers hear about an order that was rolled back.

The transactional outbox solves this by writing the event to an outbox table in the same transaction as the business change. A separate relay reads unpublished rows and pushes them to the broker, marking them as sent. If the relay crashes after publishing but before marking, the event is sent again, which is fine because consumers are idempotent.

CREATE TABLE outbox (
  id            bigserial   PRIMARY KEY,
  aggregate_id  uuid        NOT NULL,
  event_type    text        NOT NULL,
  payload       jsonb       NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  published_at  timestamptz
);

-- Relay: claim a batch without blocking other relay workers.
WITH batch AS (
  SELECT id FROM outbox
  WHERE published_at IS NULL
  ORDER BY id
  LIMIT 100
  FOR UPDATE SKIP LOCKED
)
UPDATE outbox o
SET published_at = now()
FROM batch
WHERE o.id = batch.id
RETURNING o.id, o.event_type, o.payload;

Run that claim query inside a transaction, publish the returned rows, and commit only after the broker acknowledges. FOR UPDATE SKIP LOCKED lets several relay workers run without stepping on each other. For higher volumes, change data capture tools can tail the Postgres write-ahead log instead of polling, but the polling relay is simple and works well for most teams.

Testing that it actually works

Idempotency bugs hide in concurrency, so happy-path tests will not find them. Fire the same request in parallel from a test and assert that exactly one row was created and every response is identical. Kill the process between steps in an integration environment. Replay a captured webhook payload twice. These tests are cheap to write and tend to catch real bugs the first time they run.

Key takeaways

• Assume every request and every message will arrive more than once.
• Use an Idempotency-Key header scoped to the caller, store a request fingerprint, and handle in-flight duplicates explicitly.
• Make the deduplication record and the state change commit in one transaction.
• Encode natural uniqueness as Postgres constraints and use ON CONFLICT instead of check-then-insert.
• For webhooks: verify, dedupe by event id, acknowledge fast, process asynchronously, and do not trust ordering.
• Publish events through a transactional outbox so state and messages never disagree.

If you are building payments, order flows or integrations where duplicates cost real money, our backend team designs these guarantees in from day one. Get in touch to talk through your architecture.