Engineering notes
Why we built our own rendering engine
One warm browser, a clean slate for every page, and the numbers that made the decision.
By Roger Campos · Last updated: September 2026
TL;DR
URLpipe renders every page with its own engine: one warm Chrome, driven over the DevTools Protocol, with a fresh browser context for each visit. Against the off-the-shelf engine it replaced, on the same machine, a visit got slightly faster, six concurrent visits got 17% faster, idle memory fell from 690 MiB to 225 MiB, and a long soak showed no drift.
Free plan, no credit card. 1,000 credits a month.
Every URLpipe result starts with loading a page in a real browser. Until September 2026 that was an off-the-shelf headless-browser service. It worked, and then it became the part of the system we understood least and spent the most memory and upkeep on. This note is about what replaced it and why.
What we had
The problem with a browser per request
The previous engine launched a whole Chrome for every request. That is the simplest correct design — nothing can leak between two customers' visits if they never share a process — and it has three costs that grow with traffic: startup time on every request, memory for every browser alive at once, and cleanup that has to be perfect. It wasn't: scratch directories accumulated on disk until something noticed.
It was also a black box at exactly the layer where we needed to make decisions: what the browser reports about itself, how a challenge page is waited through, what counts as "the page has finished loading". Each of those shows up in a customer's result.
What we built
The design
- One warm Chrome. Launched once and kept running, so no request pays for startup.
- A fresh browser context per visit. A context is Chrome's own isolation unit — its own cookies, storage and cache — created for the visit and thrown away after it. Two customers' visits never share state, without paying for two browsers.
- The DevTools Protocol over a pipe, not a port. The engine talks to Chrome through a pipe, so there is no debugging port for a page to reach.
- About 900 lines of Node, with no npm dependencies. Small enough to read in an afternoon, with nothing underneath it to audit or upgrade.
- One request, one visit. The engine takes a list of captures — HTML, screenshot, console — and returns them all from the same page load. A single
/htmlis that request asking for one capture; a/scrapeis the same request asking for several.
Lighthouse audits run on a separate, equally small service with a pool of pre-launched Chromes, each serving exactly one audit and then discarded — an audit must own its browser, and measures its own page load, so it never shares one with a visit.
What the engine reports about itself is its own business and we don't publish it, beyond this: it is an ordinary, current Chrome build, pinned so that a result doesn't change underneath you on a security update, and it adds nothing to a page unless a capture or an option you asked for needs it.
Measured
The numbers that decided it
Both engines were driven the way the app drives them, sequentially on the same machine — two engines rendering at once measure the contention, not the engines.
| Previous engine | Our engine | |
|---|---|---|
| One HTML visit | 1,702 ms | 1,600 ms |
| Six concurrent visits | 2,384 ms | 1,973 ms |
| Idle memory | 690 MiB | 225 MiB |
| Throughput at 12 concurrent, second host | 4.67 req/s | 5.91 req/s |
| CPU per request, second host | 1.90 core-s | 1.26 core-s |
| 40-visit soak | — | no drift |
A single visit is dominated by the page itself, so the gain there is small. The gains that matter are under load: a third of the memory, a third less CPU per request, and a quarter more throughput from the same machine. Those are what set how many requests we can run in parallel for the same money — which is what the plans' parallel requests sell.
The trade
What got harder
Owning the engine means owning its failures. Three gates are checked in and re-run whenever the engine changes: an identity check that the browser's reports agree with each other, a sweep of real sites, and the benchmark above. A new Chrome version is a deliberate change with those gates in front of it, not something that arrives with a base image.
Two behaviours changed with the switch and are documented rather than hidden: console messages from child frames are not collected, and messages logged before a page redirects itself are not kept, because the buffer belongs to the document that logged them.
And owning it made it possible to keep a promise the old setup never kept: screenshots are full page, scrolled first so lazy-loaded images appear.
Engineering notes
More engineering notes
One page visit, several answers
How /scrape reads several results off one page load and stays identical to calling each endpoint on its own.
Pacing a site that can't keep up
When a target site falls over under load, slow down and try again — without failing the request or billing the wait.
Why async is the default
Pages take seconds to minutes and sometimes much longer; a request that returns a token at once never loses the answer.
Billing only what worked
Failures, bot checks, duplicates, cache hits and waits are free — and the harder case, a 200 that is really a block page.
Request timing as marks, not spans
Store four moments per request and derive every duration from them — and publish exactly one number.
FAQ
Frequently asked questions
Does URLpipe use a real browser?
Can one customer's visit see another's cookies?
Which Chrome version renders my pages?
Is Lighthouse run in the same browser?
Turn any URL into clean data in minutes.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.