Engineering notes
One page visit, several answers
The page load is the slow part. Do it once, and make sure nothing else changes.
By Roger Campos · Last updated: September 2026
TL;DR
/scrape loads a page once and reads every requested result off that visit: the HTML, the screenshot and the console output come from the capture, and Markdown, metadata, summary and keywords are derived from the HTML. It stays identical to the single endpoints because it is not a second implementation — each result is produced, recorded and stored exactly as its own endpoint would, under the same key.
Free plan, no credit card. 1,000 credits a month.
Every analysis starts by loading the page in a browser, and that load is most of the time a request takes: a median of 3.5 seconds for /html. Ask for Markdown, metadata and a screenshot as three calls and you pay that three times. /scrape pays it once.
The hard part is not doing it once. It is making sure the results are the same ones the three calls would have given you — because if a scrape's Markdown differed from /markdown's, every cache, every comparison and every test you wrote against one would be wrong for the other.
The shape
One visit, three captures, five derivations
The rendering engine takes a list of captures — HTML, screenshot, console — and returns them from one page load. That is the only interface it has: a single /html is a visit asking for one capture, and a scrape is the same visit asking for several. There is no separate "scrape mode" in the browser to drift from the single one.
Everything else is derived from the HTML: html itself, markdown, and the three AI extractions. The extractions run in parallel, bounded so one scrape can't hold more than its share of database connections, and each reads its own slice of the page exactly as its endpoint does.
Two things take a visit of their own, on purpose. /lighthouse measures a page load, so it can't share someone else's; it is queued first, on its own lane, and awaited at the end. And a screenshot with an option that changes how the page loads — viewport, scale factor, dark mode — gets its own load, so the HTML and Markdown beside it stay the page as it normally renders.
The rule
Why it stays identical
A scrape is recorded as one parent request and one child per operation. Each child is finished exactly as an individual request would be — the same success and error handling, the same timing marks, and the result stored under the same key the individual endpoint uses. So a scrape can be answered from a /markdown you made yesterday, and a /meta tomorrow can be answered from today's scrape, both free.
That rule is also what keeps billing boring. Credits are counted from the children, by operation, with the same query that counts single calls; a scrape costs the sum of its operations, and a cached operation inside it costs nothing. The one place a scrape is cheaper is a residential fetch, charged per visit rather than per operation, because the bandwidth really is spent once.
The tests assert it from both directions: a scrape's results are served to the individual endpoints and the other way round, a fresh individual result is served to a scrape without fetching, and on the free plan a scrape the remaining allowance can't cover in full is refused whole, before any work.
Partial results
Failing one at a time
- An operation that fails — an AI extraction on a page too large, a screenshot selector that matched nothing — fails alone. Its siblings still answer, each with its own
successflag. - A failure of the page itself — it doesn't resolve, it answers 404, robots.txt disallows it — fails every operation that would have shared the visit, and costs nothing.
- A target host too busy to serve the page sends the whole scrape back to the queue to be paced, rather than failing each operation; results already served from cache keep their answers.
- Operations come back in the order you asked for them.
A scrape counts as one request against your plan's parallel limit however many operations it runs, because it is one page visit.
curl -X POST https://urlpipe.dev/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "operations": ["markdown", "meta", "screenshot"], "sync": true}'Engineering notes
More engineering notes
Why we built our own rendering engine
One warm Chrome, a fresh browser context per visit, about 900 lines with no dependencies — and the measurements that decided it.
Pacing a site that can't keep up
When a target site falls over under load, slow down and try again — without failing the request or billing the wait.
Why async is the default
Pages take seconds to minutes and sometimes much longer; a request that returns a token at once never loses the answer.
Billing only what worked
Failures, bot checks, duplicates, cache hits and waits are free — and the harder case, a 200 that is really a block page.
Request timing as marks, not spans
Store four moments per request and derive every duration from them — and publish exactly one number.
FAQ
Frequently asked questions
Is a /scrape result the same as calling the endpoint directly?
Does /scrape cost less than separate calls?
Why does Lighthouse run separately inside a scrape?
What happens when one operation fails?
Turn any URL into clean data in minutes.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.