Glossary
Bot detection
The gatekeeping in front of much of the web, described from the side of the one being checked.
By Roger Campos · Last updated: September 2026
TL;DR
Bot detection is the set of checks a site, or the service in front of it, runs to decide whether a visitor is a person or automated software. It reads signals such as the IP address's network, the client's protocol behaviour and whether JavaScript runs, and can let the request through, challenge it, or block it.
Free plan, no credit card. 1,000 credits a month.
How it works
What sites look at
Most large sites sit behind a CDN or security service that scores every request before the site sees it. The broad families of signal are public: the reputation of the IP address and whether it belongs to a datacentre; how the client speaks HTTP and TLS; whether the client runs JavaScript and what the environment it reports looks like; and how requests are paced. No single signal decides, and the thresholds are each site's own.
The outcomes are three. The request passes. The request gets a challenge — an interstitial page whose script checks the browser and then reloads the real page, sometimes with a puzzle for a human. Or it is blocked, often with a 403.
In practice
Why a real browser matters
A plain HTTP client cannot run a challenge's script, so it never gets past the interstitial: curl comes back with a page that says "checking your browser" and nothing else. A real browser runs it like any visitor's browser would, and many challenges clear on their own. That is also the reason not to judge a fetch by its status code — a challenge page can come back as 200 with none of the content in it.
What a real browser does not give you is a right to the page. If a site has decided not to serve automated visitors, the respectful answer is to take that as its answer.
URLpipe
How URLpipe handles a bot check
Pages load in real Chrome. When a site answers with a challenge, the engine recognises it, waits for it to clear, and retries before you hear about it. When it still does not clear, the request fails with The page asked us to complete a bot check before it would load. and costs nothing — a bot check we could not clear is never billed. For sites that treat datacentre addresses differently, a residential exit changes the one signal most sites act on first. It is not a guarantee, and nothing is.
FAQ
Frequently asked questions
Why do I get a 403 when scraping a site?
What is a JavaScript challenge?
Does URLpipe charge for pages blocked by a bot check?
See it on your own pages.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.