Async & sync modes

By default, URLpipe processes every request asynchronously — it accepts the request instantly and hands you a token. Collect the result at a webhook we POST to, or fetch it by that token whenever you like. Add sync: true to any request to get the result back inline in the HTTP response instead.

Choosing a mode

Pick the mode per request with the sync parameter. Omit it (or send sync=false) for async — the default, which suits long-running work like /lighthouse and high-volume batches, since you never hold a connection open. Send sync=true for sync — simplest for quick, interactive calls where you want the answer right away.

Typical response times

How long a request takes depends mostly on the target page — its weight, how much JavaScript it runs, and how fast its own server responds. The figures below are drawn from production traffic for non-cached requests: p50 is the median and p90 is the 90th percentile, so 9 in 10 requests finish at or under it.

Async requests

Async is the default, so no sync parameter is needed. The endpoint responds immediately with a token, and the result reaches you one of two ways: we POST it to your webhook, or you fetch it with GET /result/:token when you are ready. Both are always available — the token in this response is what makes the second one work, whether or not a webhook is configured.

That immediate response also carries the usual metadata headers: the token, whether the result was already cached (and how old it is), and your remaining quota. The one it can't carry is X-Processing-Time-Ms — it measures how long the work took, and on a cache miss the work hasn't started. The number arrives with the result, in the webhook's meta object or on the GET /result/:token response. On a cache hit there was nothing to run, so the accept carries it already.

Where results go

Add report_to to a request to have that result delivered there. Set a default endpoint for the project (Settings → Default Webhook Endpoint) to have every async result delivered without repeating the URL on each call. Both are optional, and the request wins over the project:

A request that names no endpoint and belongs to a project with no default is still accepted and processed — it simply produces no webhook. That is the right shape if you poll, or if you queue tokens and collect results in your own time. Naming an endpoint we cannot deliver to, on the other hand, is a 422: it is a mistake worth reporting, whereas naming none is a choice.

Which endpoints we accept

A webhook endpoint is held to the same rules as the URL you ask us to analyse — both are requests we make from our own infrastructure. It must be a public http/https address on the scheme's default port, at a real domain name.

• No IP addresses, in any notation — including decimal, octal, hex, short forms and IPv6
• No localhost, bare hostnames, or internal suffixes like .local and .internal
• No credentials in the URL — we would store them and show them back to you
• No custom ports: https://hooks.example.com:8443/x is refused, https://hooks.example.com/x is fine

This applies to the report_to parameter and to the project default alike, and it is enforced again at delivery time — so an endpoint that stops qualifying is refused rather than called. A refused delivery is recorded on the Webhooks page with the reason, and the result stays available from GET /result/:token.

curl -X POST https://urlpipe.dev/markdown \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "report_to": "https://your-app.com/webhooks/urlpipe"
  }'

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
X-Result-Token: 0Zx3…9aQ
X-Cache: miss
X-Quota-Category: ai
X-Quota-Limit: 50
X-Quota-Remaining: 43
X-Quota-Reset: 2026-08-31T23:59:59Z

{
  "token": "0Zx3…9aQ",
  "status": "accepted"
}

The webhook delivery

When processing finishes, URLpipe sends a POST with a JSON body to the endpoint the request resolved to. The token matches the one from the immediate response, so you can pair it with your original request. A request with no endpoint skips this entirely; the same fields come back from GET /result/:token instead.

{
  "token": "0Zx3…9aQ",
  "operation": "markdown",
  "success": true,
  "result": "# Example Domain\n\nThis domain is for use…",
  "error": null,
  "meta": {
    "cache": "miss",
    "cache_age": null,
    "processing_time_ms": 4182,
    "quota": {
      "category": "ai",
      "limit": 50,
      "remaining": 43,
      "resets_at": "2026-08-31T23:59:59Z"
    }
  }
}

{
  "token": "0Zx3…9aQ",
  "operation": "markdown",
  "success": false,
  "result": null,
  "error": "The request timed out.",
  "meta": {
    "cache": "miss",
    "cache_age": null,
    "processing_time_ms": 60104,
    "quota": {
      "category": "ai",
      "limit": 50,
      "remaining": 44,
      "resets_at": "2026-08-31T23:59:59Z"
    }
  }
}

Signing & verifying webhooks

Your report_to URL has to accept an unauthenticated POST from the internet. Turn on webhook signing and every delivery carries an HMAC signature instead, so your endpoint can prove the request came from URLpipe and not from someone who learned the URL.

It is off by default and enabled per project under Settings → Webhook Signing in your dashboard, which is also where the signing secret appears once it is on (it starts with whsec_). One secret per project, used for every endpoint's deliveries.

POST /webhooks/urlpipe HTTP/1.1
Content-Type: application/json
X-URLpipe-Timestamp: 1756568400
X-URLpipe-Signature: v1=3a1f9c…7b2e

signed = "1756568400" + "." + raw_body
signature = HMAC_SHA256(whsec_…, signed)

import hashlib, hmac, json, os, time
from flask import Flask, request, abort

TOLERANCE = 5 * 60
SECRET = os.environ["URLPIPE_WEBHOOK_SECRET"].encode()

@app.post("/webhooks/urlpipe")
def urlpipe_webhook():
    # request.get_data() is the raw body; request.json is not.
    body = request.get_data()
    timestamp = request.headers.get("X-URLpipe-Timestamp", "")
    header = request.headers.get("X-URLpipe-Signature", "")

    # While rolling verification out, before signing is enabled,
    # accept a delivery that carries no signature:
    #   if not header: return process(body)
    if not header or not timestamp.isdigit():
        abort(401)
    if abs(time.time() - int(timestamp)) > TOLERANCE:
        abort(401)

    signed = timestamp.encode() + b"." + body
    expected = "v1=" + hmac.new(SECRET, signed, hashlib.sha256).hexdigest()

    # Any one of the comma-separated signatures may match: during a
    # rotation we send the new and the old.
    if not any(hmac.compare_digest(s.strip(), expected) for s in header.split(",")):
        abort(401)

    return process(body)

def process(body):
    enqueue(json.loads(body)["token"])
    return "", 200

const crypto = require("crypto");
const TOLERANCE = 5 * 60;

const matches = (header, expected) =>
  header.split(",").some((signature) => {
    const a = Buffer.from(signature.trim());
    const b = Buffer.from(expected);
    return a.length === b.length && crypto.timingSafeEqual(a, b);
  });

// express.raw() so req.body is a Buffer, not a parsed object.
app.post("/webhooks/urlpipe", express.raw({ type: "application/json" }), (req, res) => {
  const timestamp = Number(req.get("X-URLpipe-Timestamp"));
  const header = req.get("X-URLpipe-Signature");

  // While rolling verification out, before signing is enabled,
  // accept a delivery that carries no signature:
  //   if (!header) return process(req.body, res);
  if (!header || !Number.isFinite(timestamp) ||
      Math.abs(Date.now() / 1000 - timestamp) > TOLERANCE) {
    return res.sendStatus(401);
  }

  const expected = "v1=" + crypto
    .createHmac("sha256", process.env.URLPIPE_WEBHOOK_SECRET)
    .update(`${timestamp}.${req.body}`)
    .digest("hex");

  // Any one of the comma-separated signatures may match: during a
  // rotation we send the new and the old.
  if (!matches(header, expected)) return res.sendStatus(401);

  process(req.body, res);
});

function process(body, res) {
  enqueue(JSON.parse(body).token);
  res.sendStatus(200);
}

$tolerance = 5 * 60;
$secret = getenv("URLPIPE_WEBHOOK_SECRET");

// php://input is the raw body; $_POST is not.
$body = file_get_contents("php://input");
$timestamp = $_SERVER["HTTP_X_URLPIPE_TIMESTAMP"] ?? "";
$header = $_SERVER["HTTP_X_URLPIPE_SIGNATURE"] ?? "";

// While rolling verification out, before signing is enabled,
// accept a delivery that carries no signature:
//   if ($header === "") { process($body); exit; }
if ($header === "" || !ctype_digit($timestamp) ||
    abs(time() - (int) $timestamp) > $tolerance) {
    http_response_code(401);
    exit;
}

$expected = "v1=" . hash_hmac("sha256", "{$timestamp}.{$body}", $secret);

// Any one of the comma-separated signatures may match: during a
// rotation we send the new and the old. hash_equals is the
// constant-time compare — never === on a secret-derived value.
$matched = false;
foreach (explode(",", $header) as $signature) {
    if (hash_equals($expected, trim($signature))) {
        $matched = true;
    }
}
if (!$matched) {
    http_response_code(401);
    exit;
}

enqueue(json_decode($body, true)["token"]);
http_response_code(200);

Turning signing on and off

Both take effect on the next delivery, including a retry of a webhook whose earlier attempts went out the other way. The two directions are not equally safe.

Turning it on is safe at any time. The signature rides in headers and never changes the payload — same method, same Content-Type, byte-identical body — so an endpoint that ignores the headers is unaffected. Enable it first, then add verification whenever you are ready. Doing it the other way round means your endpoint rejects unsigned deliveries before we sign any, so keep the no signature branch in the examples below until signing is on.

Turning it off is the breaking direction. Deliveries stop carrying the headers at once, so an endpoint that requires a signature will reject every result. Remove or relax your verification before you disable it.

Rotating the secret

Rotating in Settings opens a 24-hour grace window rather than cutting over. For those 24 hours every delivery carries two signatures — the new secret first, then the one it replaced — so an endpoint still holding the old secret keeps verifying while you deploy the new one, in either order and with nothing in flight rejected. After the window, only the new secret is sent.

That window is not what you want for a leaked secret: signing with the old key is exactly what keeps a leaked key useful to a forger, for as long as your endpoint honours it. Rotate, deploy the new secret, then use Stop accepting old secret in Settings to end the window immediately.

Delivery guarantees

• A result that resolved to an endpoint is delivered there as a JSON POST. One that resolved to none is not delivered at all, and waits for you at GET /result/:token
• Webhook delivery uses a 30-second timeout.
• If delivery fails, the result is still stored and remains visible in your project's history in the dashboard, so nothing is lost.
• Each request gets a unique token, so concurrent jobs never collide.
• With signing enabled, every delivery — and every retry of one — is signed with your project's secret, carrying a fresh timestamp, so a retry passes a freshness check like any other delivery. With it off, no signature headers are sent.
• A failed delivery can be sent again by hand from the Webhooks page in your dashboard, at any time — there is no window on it, which matters because the automatic attempts are long finished by the time an endpoint is fixed.
• A delivery your endpoint rejects — a failed signature check included — is retried with backoff, up to six attempts in all spread over roughly twenty minutes. After that we stop and email your organization's admins. The result is not lost either way: it stays in your history and behind GET /result/:token for the full retention window.
• The meta object is additive — every other payload field keeps its name, type and meaning, so existing consumers need no changes.
• The figures in meta.quota are taken when the webhook is delivered, not when the analysis finished, so a retried delivery reports your allowance as of that attempt.
• Prefer to pull instead of receive a webhook? Fetch the result any time with GET /result/:token.

Sync mode

Send sync=true to process a request synchronously. Webhooks play no part: report_to and the project's default endpoint are both ignored, because the response IS the delivery. The request is processed inline and the result comes straight back in the HTTP response body, in the content type documented for each endpoint. Failures return a 422 with an error field.

Every response — sync or async — carries an X-Result-Token header identifying the request, so you can re-fetch the result later via GET /result/:token. It arrives alongside the cache status, the processing time and your remaining quota: see Response headers.

Sync timeouts

A sync request waits up to 60 seconds for the analysis to finish. If it isn't ready in time, the endpoint returns 504 Gateway Timeout with the token in the body — but the analysis keeps running in the background. Retrieve the result once it completes via GET /result/:token, or use async mode (omit sync) for consistently long-running work.

{
  "error": "processing_timeout",
  "message": "The analysis is taking longer than expected. Retrieve it later via GET /result/:token, or use async mode for long operations.",
  "token": "0Zx3…9aQ"
}