Changelog · September 24, 2026
Each project chooses whether to follow robots.txt
Follow each site's crawl rules by default, or turn them off for a project where you have the right to fetch.
By Roger Campos · Last updated: September 2026
TL;DR
Every project now has a robots.txt setting, on by default. Before any page visit — every endpoint, /scrape and the MCP tools included — URLpipe reads the site's robots.txt and does not fetch a page it disallows. The request fails with a 422 naming the reason and costs nothing.
Free plan, no credit card. 1,000 credits a month.
Projects gained a robots.txt card in Settings. With it on — the default — URLpipe reads the target site's robots.txt before each page visit and refuses a page it disallows, for every endpoint, /lighthouse, /scrape and MCP. The refusal is a 422 that says why, and like every failed request it spends no credits and is not retried.
Parsing follows RFC 9309: the URLpipe group if the file has one, else *; the longest matching rule wins, with allow on a tie; * and a trailing $ work as wildcards. The file is cached for 24 hours. A file we can't read (a 5xx, a timeout) allows the fetch — only a rule the site wrote refuses a page.
When you own the site
Turning it off
Switch the setting off for a project and its requests fetch pages whatever robots.txt says; the file is never read. That is for sites you own or have permission to fetch — with the setting off, you are responsible for having the right to fetch those pages.
A result fetched with the setting off is never served from the store to a project that follows robots.txt, so the two kinds of project never share an answer.
Endpoints
The endpoints on this page
- Get the rendered HTML of any URL
Paste a link and get the page's HTML after JavaScript has run and redirects have been followed — the DOM a real browser sees, not the empty shell curl returns.
Try it free - Turn any URL into clean Markdown
Paste a link and get the page's main content as tidy Markdown — headings, lists, links and code kept, navigation and cookie banners stripped. The format LLMs and RAG pipelines work best with.
Try it free
FAQ
Frequently asked questions
Is robots.txt followed by default?
Am I charged for a page robots.txt refused?
Which user agent group does URLpipe read?
Try it on the free plan.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.