Engineering notes
Pacing a site that can't keep up
A crawl of a small shop on shared hosting should finish slowly, not fail 24 times in a row.
By Roger Campos · Last updated: September 2026
TL;DR
When a site times out, returns a 5xx or says 429, URLpipe climbs a per-host ladder of 0, 2, 5, 15, 30 and 60 seconds between requests to it, and retries instead of failing; every page served steps back down a rung. Requests keep trying for 10 minutes from when they were accepted. A healthy host costs one index lookup that finds nothing, and nobody is billed for the wait.
Free plan, no credit card. 1,000 credits a month.
What happened
The failure
A customer pointed a crawl at a small online shop. Seven pages rendered fine in thirty seconds. Then pages took seven seconds, then the navigation budget started expiring, and for the next four and a half minutes 24 requests in a row failed with a timeout. Two minutes later the shop was fine again: the same pages rendered in 3.3 seconds, eight times out of eight.
Nothing was wrong with the pages or the network. The shop's server pool was full — of our own traffic. We were running up to five renders at once against one host, each costing it several requests plus its assets, and once it fell over we kept knocking and reporting each rebuffed knock to the customer as a failed request.
Two things were wrong on our side. The pace was fixed in advance, one number for every host on the internet — a shop on shared hosting and a news site behind a CDN were the same to it. And a busy host was treated as an answer: "the request timed out" was written, delivered and counted, for a page that was available a minute later.
The pace
A ladder per host
Each host that is currently misbehaving gets one row, holding a rung on a ladder of seconds between requests:
LADDER = [0, 2, 5, 15, 30, 60] # seconds between requests to one host, by rungEvery stall climbs a rung; every page served steps back down one; the bottom rung deletes the row. Up fast, down slowly, for the usual reason: overshooting a struggling origin costs much more than being slow to speed back up, and the first page back after an outage is evidence the origin answered once, not that it has recovered.
A healthy host — every host, until one gives us a reason — has no row, costs one indexed lookup that finds nothing, and behaves exactly as if none of this existed.
What counts
Only the failures worth waiting out
| Failure | Why it is worth waiting out |
|---|---|
| Timeout | The commonest shape: a full server accepts the connection and says nothing |
| Connection timeout | The connection could not be set up in time |
| The target's 5xx | Including the 503 a server sends when it is out of capacity |
| 429 Too Many Requests | The site said so outright |
Everything else is answered at once. A refused connection, a name that doesn't resolve, a bad certificate, a 404 — those are stable, and retrying them for ten minutes would only delay an honest answer the caller could act on. A 429 gets a kind of its own for the same reason: reported as a bad request, it would tell the caller their request was wrong and tell the pacing nothing — ignoring the one explicit "slow down" a well-behaved site gives.
Always an answer
Bounded by time, not attempts
Retries run inside a 10-minute window measured from when the request was accepted. Once it closes, pacing doesn't engage at all and the request answers with what it gets — which is what guarantees every request terminates. A count of attempts couldn't do that: it can't tell a wait from a failure, and a request held four times by the pacer hasn't failed at anything. The Playground and the free tools get 45 seconds instead, because a person is watching a spinner.
A discarded attempt leaves nothing behind — no failure, no timing marks — so the request's reported duration covers the attempt that answered plus everything it waited for, and nothing is reported twice.
A livelock worth knowing
Why waiters don't hold reservations
The obvious design is for each waiting request to book the next free slot: FIFO order, perfect spacing. It livelocks. A request that has to wait books a slot, sleeps, wakes, asks again — and books itself a new slot behind everything that arrived while it slept. Every wake-up pushes every waiter back, and the queue for one host never drains.
So a claim only succeeds if the current slot is free, as one conditional update, and waiters re-ask instead of reserving. That gives up FIFO order and buys the only property that matters: the queue drains at exactly one request per pace.
Who pays
Not billing the wait
The pace protects someone else's server, so it must not be charged to you. A request waiting on a paced host spends no credits until it produces a result, and it doesn't hold one of your plan's parallel slots while it waits — otherwise traffic from someone else that knocked a host over could lock a Free plan, with one slot, out of every URL. What you see is a slower request, never a failed or a dearer one.
Engineering notes
More engineering notes
Why we built our own rendering engine
One warm Chrome, a fresh browser context per visit, about 900 lines with no dependencies — and the measurements that decided it.
One page visit, several answers
How /scrape reads several results off one page load and stays identical to calling each endpoint on its own.
Why async is the default
Pages take seconds to minutes and sometimes much longer; a request that returns a token at once never loses the answer.
Billing only what worked
Failures, bot checks, duplicates, cache hits and waits are free — and the harder case, a 200 that is really a block page.
Request timing as marks, not spans
Store four moments per request and derive every duration from them — and publish exactly one number.
FAQ
Frequently asked questions
Why did my request take longer than usual?
Which errors does URLpipe retry automatically?
How long does URLpipe keep retrying?
Am I charged for pacing?
Turn any URL into clean data in minutes.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.