Research
What stops a headless browser in 2026
A sweep of sites known for bot protection, from two kinds of address — and the one kind of refusal that ends up on your invoice.
By Roger Campos · Last updated: September 2026
TL;DR
Of 46 well-known, frequently protected sites visited by a real, honestly configured Chrome, 39 served the page from both a datacentre and a residential address. Three refused only the datacentre address; two refused both; one answered 200 with a block page; one throttled whichever address visited too often. The 200 block page is the class that matters, because an API that trusts the status code bills it as a page.
Free plan, no credit card. 1,000 credits a month.
Summary
The results
Last checked 2026-09-21. Each site was visited from a datacentre address and from a residential one.
| What happened | Sites | What the caller should get |
|---|---|---|
| Served the real page from both addresses | 39 | The page |
| Refused the datacentre address, served the residential one | 3 | A refusal — or the page, with a residential exit |
| Refused from both addresses | 2 | A refusal |
| Answered 200 with a block page | 1 | A refusal — but the status says success |
| Served at first, then throttled the address after repeat visits | 1 | A refusal, until the address cools down |
The list is not a random sample of the web. It was built from sites known for bot protection — large retail, travel, classifieds, social networks, ticketing. Most of the web does not fight back at all; this is the part that does. We don't publish which site is in which row: the point is the classes, and naming the sites would be discourteous to them.
The majority
Most protected sites serve a well-behaved browser
39 of 46 served the real document from both addresses. Several of them run a check inside the page before serving it — a script that decides whether the visitor is a browser — and a real Chrome that doesn't misreport what it is passes them. Two of the 39 were answering with a challenge page carrying a 200 or 202 status rather than a 4xx; once the engine read the page instead of the status, both cleared in four to seven seconds.
How the engine is configured is not something we publish. What matters for this study is that it is an ordinary, current Chrome, loading each page in a fresh browser context, and that every result below was checked by looking at what came back rather than at the status code.
The billing lesson
The 200 that is really a block page
A refusal with a 4xx is easy: the status says it failed, and nobody should pay for it. The expensive class answers 200 — sometimes 202 — with a document that is a block page or a challenge. An API that trusts the status reports success, stores the block page as the result, and bills for it.
We found this in our own product before any customer reported it. Two sites were answering with a challenge page and a 200 or 202, and we were capturing the challenge and counting it as a page: on one, a result of one byte; through the API, a Markdown document whose whole body was the page's title. The screenshot of the same visit showed the site rendered, which is what said the fault was ours.
- A challenge we recognise is waited through, whatever status it arrives with. If it doesn't clear, the request fails as
challengedand costs nothing. - A document with no body is refused rather than stored as a page.
- A failed request is never billed — a bot check we couldn't clear included.
Not solved everywhere
One site in the sweep answers 200 with its own plain block page — a normal-sized document whose whole text says the visitor is blocked, with no marker that identifies it as a challenge. That page is not yet told apart from a real one, so it is returned as a result, and billed. If a result's text is a sentence about being blocked, it is; tell us and it goes on the list.
If you run your own headless browser, the check is cheap: before trusting a 200, look at the document. An empty body, a <head> with nothing after it, or a couple of kilobytes of markup on a page that should be large is a block or a challenge, never a page. A screenshot of the same visit settles it.
Datacentre vs home
What a residential exit changes
Three sites refused the datacentre address and served the residential one, with no change to the browser at all. Those decisions are made on the address — a plain HTTP client on the same datacentre machine was refused too. No change to the browser fixes that; a different address does.
That is what URLpipe's residential: true is for. It costs 25 extra credits per page visit, and if no residential exit can take the request it fails unbilled rather than quietly falling back to the datacentre.
And one case where a residential address makes things worse: a site that served both addresses at first began throttling whichever one had visited most, answering it with a challenge while the other address was still served. One home address is a much smaller budget than a datacentre range, so spending it on repeated visits exhausts it sooner. The answer to throttling is pacing the requests, not changing the address.
The rest
Refused from everywhere
Two sites refused both addresses. One serves a challenge that never completes, whatever waits for it; the request fails after three attempts, about 29 seconds, and is not billed. The other refused one particular form of URL on 7 of 9 visits while serving the same page at a slightly different address every time — the refusal came from a redirect inside the visit, not from anything about the browser.
How it was measured
Method
- 46 sites, visited by the engine URLpipe uses in production — a pinned Chrome build, one fresh browser context per visit.
- Each site visited from two exits: a datacentre address in Germany and a residential address in Spain. Where the two answered differently, both answers are recorded.
- Sites that behaved inconsistently were visited several times (up to nine) to separate a pattern from a bad moment.
- A result is "served" when the stored document is the real page — checked by its size, its text and a screenshot from the same visit — not when the status is 200.
- This round is from 2026-09-21. Sites change their defences; a re-run will not match it exactly.
Endpoints
The endpoints on this page
- Get the rendered HTML of any URL
Paste a link and get the page's HTML after JavaScript has run and redirects have been followed — the DOM a real browser sees, not the empty shell curl returns.
Try it free - Turn any URL into clean Markdown
Paste a link and get the page's main content as tidy Markdown — headings, lists, links and code kept, navigation and cookie banners stripped. The format LLMs and RAG pipelines work best with.
Try it free
Research
More research
Web-to-Markdown, measured
Four HTML-to-Markdown strategies on the same 33 real pages: how much of the visible text each keeps, and how much of its output the reader never saw.
Lighthouse variance: how much does one audit move?
Ten mobile Lighthouse audits of each of five public pages, back to back: the performance score moved by up to 61 points on the same page.
FAQ
Frequently asked questions
Can a headless browser load sites with bot protection?
Does a residential proxy help against bot protection?
Why is a 200 block page a billing problem?
Does URLpipe charge for a bot check it couldn't pass?
Which sites are in the list?
Try the converter on your own pages.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.