Engineering notes
Why async is the default
A request that holds a connection open is betting on the slowest site you'll ever ask about.
By Roger Campos · Last updated: September 2026
TL;DR
A URLpipe request answers straight away with a token by default, and the result arrives at your webhook or from GET /result/:token. Rendering a page takes seconds — a median of 3.5 s for /html and 15.3 s for /lighthouse — and a slow site can take minutes. sync: true waits up to 60 seconds for you, and a request that outlives that still isn't lost: it answers with its token.
Free plan, no credit card. 1,000 credits a month.
Most APIs answer the question in the response. Ours answers "got it — here is your token", and the result follows. That surprises people on their first call, so here is why.
The numbers
The work is slow, and unevenly so
| Endpoint | p50 | p90 |
|---|---|---|
| /html, /markdown | 3.5 s | 4.7 s |
| /screenshot | 4.0 s | 7.6 s |
| /keywords | 5.8 s | 8.1 s |
| /console | 6.2 s | 6.7 s |
| /meta | 6.3 s | 10.8 s |
| /summarize | 7.9 s | 11.6 s |
| /lighthouse | 15.3 s | 20.5 s |
Those are production figures for non-cached requests, from the async docs. The tail is the problem: a page that runs a lot of JavaScript, a server having a slow minute, a site we are pacing for up to ten minutes. A synchronous call has to guess a timeout for all of them, and every proxy, load balancer and serverless function between you and us has its own, usually shorter.
The design
What async buys
- The answer can't be lost. The token comes back before any work starts. Whatever happens to your connection, the result is collected from GET /result/:token for 30 days, or delivered to your webhook — retried, and signed if you turn signing on.
- Your side holds nothing open. A worker that fires a hundred requests and moves on uses a hundred short connections, not a hundred long ones.
- Retries are safe. A retry with the same Idempotency-Key returns the first request's token; an identical request arriving while the first runs waits for it, free.
- The same path for everyone. Every request, sync or async, goes through the same bounded job queue — sync just waits on the result for you.
The opt-in
What sync: true actually does
sync: true doesn't run the page in the web request. It queues the work exactly like an async request and holds your connection open, checking for the result, for up to 60 seconds. Every request is paced, capped per target host and counted the same way whichever mode asked — a sync caller can't get round the limits that protect the sites we fetch.
If the work outlives the wait, the response is a 504 carrying the token, and the analysis carries on. Nothing is lost and nothing is charged twice: collect it later. Cache hits are the exception: a sync cache hit answers inline in milliseconds, and an async one is finished the moment it is accepted.
curl -X POST https://urlpipe.dev/lighthouse \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
# => {"token": "0Zx3…9aQ", "status": "accepted", …}
curl https://urlpipe.dev/result/0Zx3…9aQ -H "Authorization: Bearer YOUR_API_KEY"
# 202 while it runs, 200 with the report when it's doneThe honest answer
When to choose sync
When a person or an agent is waiting on this one result and has nothing else to do — a CLI, a chat turn, a script you run by hand. Everywhere else, and always for batches, async is the better default, which is why it is the default. The MCP server follows the same rule and tells agents so in its instructions: a token is not a failure.
Engineering notes
More engineering notes
Why we built our own rendering engine
One warm Chrome, a fresh browser context per visit, about 900 lines with no dependencies — and the measurements that decided it.
One page visit, several answers
How /scrape reads several results off one page load and stays identical to calling each endpoint on its own.
Pacing a site that can't keep up
When a target site falls over under load, slow down and try again — without failing the request or billing the wait.
Billing only what worked
Failures, bot checks, duplicates, cache hits and waits are free — and the harder case, a 200 that is really a block page.
Request timing as marks, not spans
Store four moments per request and derive every duration from them — and publish exactly one number.
FAQ
Frequently asked questions
Why does URLpipe return a token instead of the result?
How long does sync: true wait?
Do I need a webhook to use async?
Is sync faster than async?
Turn any URL into clean data in minutes.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.