Check whether robots.txt lets a bot fetch a URL
Paste a link and see what the site's robots.txt says about that exact path — for URLpipe's user agent and for every crawler that has no group of its own — and which rule decides it.
- No signup
- No credit card
- The rules URLpipe itself follows
Your result will appear here.
Enter a URL above, or pick one of the example pages.
What you get
- An allowed or disallowed verdict for URLpipe and for *
- The user-agent group that applied and the rule that matched
- The robots.txt file itself, as the site serves it
How it works
- 1We fetch the site's /robots.txt from our own network, following up to 5 redirects.
- 2Rules are read the RFC 9309 way: the group naming the agent applies, else the * group; the longest matching rule wins, and an allow beats a disallow of the same length.
- 3You see the verdict for each agent, the rule that decided it and the file.
What people use it for
- Check a rule before you ship a robots.txt change
- Find out whether URLpipe will fetch a page, before you ask it to
- Debug * wildcards and $ anchors against a real path
Frequently asked
Yes, by default, on every project: before a page is visited, the site's robots.txt is checked for the URLpipe user agent, and a disallowed page is not fetched. A customer can turn that off for a project, and is then responsible for having the right to fetch those pages.
Add a group for our product token — User-agent: URLpipe — followed by Allow or Disallow lines. Without a group of its own, URLpipe follows your * group.
A missing file (a 4xx answer) means everything is allowed, as RFC 9309 says. For a file that could not be read at all — a 5xx, a timeout — the RFC tells crawlers to assume nothing is allowed; URLpipe refuses a page only when the file says so, and lets the visit itself report a site that is down.
It checks two agents: URLpipe, and * — what any crawler without a group of its own is told. Googlebot and other named bots can have their own groups; the file is shown under the verdict so you can read them.
No. It is one fetch of a small text file, not an API call.
Ready to use it in production?
Get a free API key and call every endpoint from your own code — 1,000 credits a month, no credit card.