Engineering notes
Request timing as marks, not spans
Record when things happened; compute how long they took.
By Roger Campos · Last updated: September 2026
TL;DR
Each URLpipe request stores four moments — accepted, started, finished, and first webhook attempt — and nothing else about time. Queue time, processing time and delivery time are all subtractions, so they can't drift from each other. The caller gets one number, accepted to finished, in X-Processing-Time-Ms and the webhook's meta; the breakdown is ours.
Free plan, no credit card. 1,000 credits a month.
Two questions get asked about every request, by two different people. How long did this take? is yours, and it has one answer. Where did the time go — queued, working, or stuck in our delivery lane? is ours. Timing in URLpipe is built to keep those apart: internally we account for every span; publicly we publish one number and stand behind it.
The model
Four marks
created_at ──► processing_started_at ──► finished_at ──► answered_at
queued_ms processing_ms │ delivery_ms
└──────── duration ──────────────────┘ (ours, not yours)| Mark | Meaning |
|---|---|
| created_at | The request was accepted |
| processing_started_at | Work began; the gap before it was time queued |
| finished_at | We stopped working on it |
| answered_at | The first webhook attempt, when there is one — outside the published span |
Spans are subtractions, computed on read. The one duration that is stored — because listings sort and average on it in SQL — is recomputed from the marks on every save, and assigning it directly is silently overwritten. There's a test pinning that. If you want to write a duration, you want to write a mark.
Because both spans round from the same origin and one is derived from the other by subtraction, queue plus processing equals the total exactly — rounding them separately left them a millisecond apart often enough to matter. No span is ever negative.
Published
The number you get
X-Processing-Time-Ms on a response and meta.processing_time_ms in a webhook are accepted → finished, in every mode: sync, async with a webhook, async without one. It never moves once the work is done, and a retried webhook reports the same figure as the first attempt.
An earlier design ran it to the first webhook attempt for async requests, on the reasoning that the webhook is how that caller gets the answer. Two things killed that rule. Once a webhook became optional, a request with no delivery would have had no number to report, ever. And it made one figure mean two things — how long we took, versus that plus our own delivery queue — which nobody can compare or act on.
A fact we don't have is omitted, never zeroed: an async accept on a cache miss and a sync 504 carry no processing time, because nothing has finished. A cache hit is never hardcoded to zero either — serving one is a real read, and it is timed.
Honesty
Why queue time is inside the number
Because you waited for it too. A request that sat behind others against one host took a minute from where you stand, and a "duration" that hid that would be useless for answering "why was this slow?". Requests to one host are deliberately capped and paced so we never hammer a site; under a burst, queue time is where the time goes, and it is in the number.
Nothing is carved out of it except one thing that isn't ours to count against you or against us: on a cached async request the first webhook is deliberately delayed by a couple of seconds, so your endpoint has time to store the token before the result arrives. That pause is subtracted from our internal delivery span, exactly as scheduled.
Clean records
Retries leave no trace
When an attempt is discarded — a paced host, a busy moment — its outcome and its marks are cleared with it, so the record is indistinguishable from one that has never run. The marks always describe the attempt that answered; everything before it, including our own pacing, is queue time. A /scrape's parent marks bracket every child's, so the parent's processing always covers its operations'.
Marks are written inside updates that were happening anyway — setting the status, writing the result — so timing costs no extra database writes.
Engineering notes
More engineering notes
Why we built our own rendering engine
One warm Chrome, a fresh browser context per visit, about 900 lines with no dependencies — and the measurements that decided it.
One page visit, several answers
How /scrape reads several results off one page load and stays identical to calling each endpoint on its own.
Pacing a site that can't keep up
When a target site falls over under load, slow down and try again — without failing the request or billing the wait.
Why async is the default
Pages take seconds to minutes and sometimes much longer; a request that returns a token at once never loses the answer.
Billing only what worked
Failures, bot checks, duplicates, cache hits and waits are free — and the harder case, a 200 that is really a block page.
FAQ
Frequently asked questions
What does X-Processing-Time-Ms measure?
Why is X-Processing-Time-Ms missing on some responses?
Does a retried webhook change the reported duration?
Is queue time included?
Turn any URL into clean data in minutes.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.