Engineering notes
Billing only what worked
You pay for answers, not attempts — and that is harder to guarantee than it sounds.
By Roger Campos · Last updated: September 2026
TL;DR
URLpipe charges for a request only when it produced a result. A failure of any kind, a bot check that didn't clear, a robots.txt refusal, a cache hit, a duplicate that waited for the first, an Idempotency-Key retry and the time spent waiting on a slow site all cost nothing. One query decides what is billable, for both the live allowance and the monthly invoice, so the two can't disagree.
Free plan, no credit card. 1,000 credits a month.
"We don't charge for failures" is easy to write on a pricing page and surprisingly easy to get wrong in code. Each of the cases below was either a rule from the start or a bug we found and pinned with a test.
The rule
One definition of billable
A request is billable unless it failed or was served from the store. That is one database scope, and everything that counts money reads it: the check that admits a request against your allowance, the numbers in the X-Quota-* headers, the usage screens and the overage invoice at the end of the month. When two places compute the same thing with two rules, they drift; with one, they can't.
A request still in flight counts against the allowance while it runs — otherwise a burst of parallel requests could all be admitted against the same last credits — and stops counting the moment it fails.
The list
What is never billed
| Case | Why it's free |
|---|---|
| Any failed analysis | A DNS failure, a timeout, a TLS error, an error of ours — none produced a result |
| The target's own 4xx or 5xx | It is reported as a failure with a 422, not returned as a page |
| A bot check that didn't clear | The page is there and a 403 came back — it looks almost like success, which is why it is pinned by its own tests |
| A robots.txt refusal | We chose not to fetch |
| A cache hit | No work was done |
| A duplicate that arrived mid-run | It waited on the first and was settled with its answer |
| A retry with the same Idempotency-Key | It returns the first request, charge included — once |
| A residential request with no exit free | It fails rather than falling back, and nobody pays |
| Waiting on a paced site | The pace protects someone else's server; it costs no credits and holds no parallel slot |
The bot check is the one most likely to be "fixed" into something billable by mistake, because from outside it doesn't look like a failure. So the rule is pinned for every way a caller could be charged for one: the sync response, the async record once its job has run, a scrape whose single visit was challenged, and a free-tool run.
A bug worth naming
Reporting a cost we never charged
The response headers say what a request spent. At one point that figure was computed from the operation's price rather than from the ledger — so a failed request reported a cost it had never been charged. Nobody was billed for it, but a header that says you were is almost as bad. It now derives the cost from the records under the same rule the ledger applies, so a cache hit and a failure both report nothing.
The hard case
The 200 that is really a block page
Everything above relies on knowing a request failed. The hard case is a site that refuses you with a 200: a challenge page, or a page whose whole text says you've been blocked. The status says success, so a system that trusts it stores the block page as the result and bills it.
We found two sites doing exactly that to us — answering a challenge with 200 and 202 — and we were capturing and billing the challenge: one result was a single byte; another came back through the API as a Markdown document containing only the page's title. The screenshot of the same visit showed the real site, which is what gave it away. Today a recognised challenge is waited through whatever status it arrives with, and fails unbilled if it doesn't clear; a document with no body is refused rather than stored.
It isn't solved everywhere, and we'd rather say so: a site's own plain block page, with a 200 and nothing that marks it as a challenge, is not yet told apart from a page. The study behind this has the numbers.
No surprises
Telling you as it happens
Paid plans are never cut off, so the other half of billing fairly is making sure nobody runs up an invoice without noticing: X-Quota-Cost and X-Quota-Remaining on every response, X-Quota-Overage once you are past the allowance, one email the first time each month, and a banner in the dashboard for as long as it lasts.
Engineering notes
More engineering notes
Why we built our own rendering engine
One warm Chrome, a fresh browser context per visit, about 900 lines with no dependencies — and the measurements that decided it.
One page visit, several answers
How /scrape reads several results off one page load and stays identical to calling each endpoint on its own.
Pacing a site that can't keep up
When a target site falls over under load, slow down and try again — without failing the request or billing the wait.
Why async is the default
Pages take seconds to minutes and sometimes much longer; a request that returns a token at once never loses the answer.
Request timing as marks, not spans
Store four moments per request and derive every duration from them — and publish exactly one number.
FAQ
Frequently asked questions
Does URLpipe charge for failed requests?
Am I charged if a site blocks the request with a bot check?
Is a retried request billed twice?
How do I see what a request cost?
Turn any URL into clean data in minutes.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.