Idempotency keys explained how not to pay twice for a retry

The mechanism

How an idempotency key works

1. 1
   The client generates a unique key for one logical operation — a UUID is ideal — and sends it in an Idempotency-Key header.
2. 2
   The server looks the key up. If it's new, it records the key with a fingerprint of the request, does the work, and stores the response against the key.
3. 3
   If the client retries with the same key, the server finds it and returns the stored response instead of doing the work again. One operation, one effect, one charge — however many times it was sent.
4. 4
   After a retention window the key expires, and the same value would be treated as a new request.

KEY=$(uuidgen)

curl -X POST https://api.example.com/charges \
  -H "Idempotency-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"amount": 2500, "currency": "usd"}'
# …connection reset. Retry with the SAME key:
curl -X POST https://api.example.com/charges \
  -H "Idempotency-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"amount": 2500, "currency": "usd"}'

Client side

How to use keys correctly

• One key per operation, not per attempt. Generating a fresh key on each retry defeats the purpose — every retry becomes new work.
• Create the key before the first attempt, and keep it. If the operation matters, store the key with the job or the order, so a retry after a crash or a redeploy still uses it.
• Never reuse a key for new work. The next order, the next page, the next charge each gets its own.
• Retry with backoff. A key makes retries safe; it doesn't make them free for the server. Wait longer between each.
• Know the window. A retry that arrives after the key expired is a new request. For most APIs that's a day or more, which covers any sane retry policy.

Server side

Adding idempotency keys to your own API

CREATE TABLE idempotency_keys (
  account_id    bigint      NOT NULL,
  key           text        NOT NULL,
  fingerprint   text        NOT NULL,  -- hash of method, path and body
  status        text        NOT NULL,  -- 'running' or 'done'
  response_code integer,
  response_body jsonb,
  created_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, key)
);

-- Claim the key. Zero rows inserted means someone already holds it.
INSERT INTO idempotency_keys (account_id, key, fingerprint, status)
VALUES ($1, $2, $3, 'running')
ON CONFLICT DO NOTHING;

1. 1
   Insert first. If the insert succeeds, you own the key: do the work, then store the response and mark it done.
2. 2
   If it conflicts, read the row. A different fingerprint is a client bug — refuse it. The same fingerprint and done means replay the stored response. The same fingerprint and running means wait for it, or answer 409.
3. 3
   Release keys on failures that did nothing. If the request failed validation or couldn't start, delete the row so a corrected retry isn't stuck with the error.
4. 4
   Expire old rows on a schedule — a day or two is plenty for retries.

With URLpipe

How URLpipe handles retries and keys

• Cache hits are free. A request whose result is fresher than max_age (7 days by default) is served from storage at no cost — see caching.
• Duplicates in flight share one run. An identical request that arrives while the first is still working waits for it, for free, without using one of your parallel slots.
• Idempotency-Key covers the rest. Within 24 hours, a retry with the same key returns the first request's token and result — one charge, one webhook — even with max_age=0. Replays carry Idempotent-Replayed: true.

FAQ

Frequently asked questions