robots.txt

Each project chooses whether its requests follow the robots.txt of the sites they fetch. Following it is the default for every project, and one switch in the project's Settings changes it.

The setting

• On — before a page is fetched, we read the site's robots.txt. A page it disallows is not fetched, and the request fails with an error that says why. Nothing is spent.
• Off — pages are fetched whatever robots.txt says, and the file is never read.

Find it under Settings → robots.txt in each project. It applies to every endpoint, /scrape and the MCP server included, and a request keeps the setting it was made under even if the switch changes while it is queued.

Turning it off is your call, and so is what follows from it: while it is off, having the right to fetch each page and use what comes back is your responsibility, as the Terms set out.

A disallowed page

{
  "error": "The site's robots.txt disallows this page, and this project is set to follow robots.txt."
}

The same sentence is in the webhook and in GET /result/:token for an async request. The site is never asked for the page, and like every failed analysis the request costs nothing. A /scrape fails every operation that would have read the page.

How the rules are read

The file is read the way RFC 9309 describes:

• The group for User-agent: URLpipe applies if there is one; otherwise the User-agent: * group does; otherwise nothing is disallowed.
• Rules match the page's path and query string. The longest matching rule wins, and an Allow beats a Disallow of the same length. * matches any run of characters and a trailing $ anchors the end.
• Each scheme and host has its own file: https://shop.example.com/robots.txt governs that host alone.
• A file is reused for up to 24 hours, so a change to it can take that long to apply.

Only a rule refuses a page. A site with no robots.txt, or one whose file could not be read — a server error, a timeout — has its pages fetched as usual, and if the site itself is down the request reports that instead.

Results are kept apart

A project that follows robots.txt is never handed a result fetched regardless of it. Results are shared across your organization's projects, so a page fetched by a project with the setting off is stored under its own key, and the same request from a project with it on fetches — and checks — afresh.

For site owners

Address URLpipe in your robots.txt by its product token, URLpipe:

User-agent: URLpipe
Disallow: /account/
Allow: /account/help

We read the file with this user agent:

Mozilla/5.0 (compatible; URLpipe/1.0; +https://urlpipe.dev/docs/robots-txt)