Retries & duplicates

Retry any request without paying for it twice. An identical request that arrives while the first is still running shares its work, and an Idempotency-Key turns a retry into the first request's answer.

Identical requests share one run

Send the same request again while the first is still working — a client that gave up waiting, a job that ran twice — and the second one waits for the first instead of starting over. It is free, and it does not use one of your concurrent requests.

Each request still gets its own token, labels and webhook, and each is answered with the same result — or, if the page could not be fetched, the same error, which is free for both. Its X-Cache is hit and its X-Quota-Cost is 0, exactly as for a result served from the cache.

Identical means what it means for the cache: the same operation, URL and options, from any project in your organization. A /scrape shares a run with another /scrape of the same operations and options. This happens on its own; there is nothing to turn on.

Idempotency-Key

When your connection drops before our response arrives, you cannot tell whether we received the request. Send an Idempotency-Key header, and resending it with the same key is safe: within 24 hours it returns the first request instead of starting a new one — the same token, the same result, one charge and one webhook.

# Generate one key per request, and reuse it only to retry that request
curl -X POST https://urlpipe.dev/markdown \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Idempotency-Key: 5f0c8a52-6b1e-4c3d-9a7e-2d4f1b8e9c10" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

A retry is answered in the mode it asks for: an async retry gets the token back, and a sync=true retry waits for the result — even when the first request was async. Replayed responses carry Idempotent-Replayed: true.

A key is not a cache

max_age decides how fresh a result you will accept; the key decides whether this is a request you already sent. So a retry returns the first request's result even with max_age=0: you asked for one fresh fetch, and a retry is still that one. For new work, send a new key.

When the key does not match

A key is bound to the request it first arrived with — the operation, URL, options, max_age, report_to and labels. Reusing it for a different request is refused with a 422, and nothing runs:

{
  "error": "idempotency_key_reused",
  "message": "This Idempotency-Key was sent with a different request in the last 24 hours. Use a new key for a new request."
}

A key outside the rules above is a 422 with error invalid_idempotency_key. A request refused for credits or concurrency never claims its key, so retrying it later with the same key is a new attempt.