robots.txt

The file is a list of groups. Each group names one or more user agents and gives Allow and Disallow rules for paths. A client uses the group that names its own product token, falls back to User-agent: *, and treats everything as allowed if neither exists. Under RFC 9309 the longest matching rule wins, an Allow beats a Disallow of the same length, * matches any run of characters and a trailing $ anchors the end of the path.

Here every client may fetch /account/help but nothing else under /account/, and a client identifying as URLpipe follows only its own group — so it may not fetch /search but is not bound by the * group at all. Each scheme and host has its own file: shop.example.com is governed by shop.example.com/robots.txt alone.

Every project follows robots.txt by default. Before fetching a page, URLpipe reads the site's file as URLpipe; a disallowed page is not fetched, the request fails with an error that says why, and nothing is spent. A file is reused for up to 24 hours. A site with no file, or one that errors, is fetched as usual.

Each project can turn this off under Settings. While it is off, having the right to fetch each page and use what comes back is your responsibility. Results fetched with it off are stored apart, so a project that follows robots.txt is never handed one. The details are in the robots.txt docs.