Guide

Scraping politely robots.txt, rate limits and pacing

Every page you fetch costs someone else a request. This guide covers the rules sites publish, how to read them the way the standard says, and how to pace yourself when a site starts to struggle.

By Roger Campos · Last updated: September 2026

TL;DR

Polite scraping means three things: read and follow robots.txt as RFC 9309 defines it, limit how many requests you send each host at once, and slow down when a host shows strain — timeouts, 429s, 503s. robots.txt is a request, not an access control; honouring it, and fetching only pages you have the right to use, is the fetcher's responsibility.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The principle

Every request you send costs someone else

A page you fetch is served by somebody's server, on somebody's bill. Politeness is keeping that cost small and respecting the rules the site publishes — and it's also what keeps you from being blocked.

Three habits cover almost all of it: read the site's robots.txt and follow it, don't send any one host more requests at once than it can comfortably serve, and slow down the moment it shows strain. None of them needs a big system — a parser, a per-host limit and a backoff rule.

One boundary first. robots.txt tells automated clients what the site would rather they didn't fetch. It is a published request, not a lock and not a licence: whether you may fetch a page and use what's on it also depends on the site's terms, copyright and the law where you operate. Following robots.txt is the baseline, and the rest is on whoever does the fetching.

The standard

What RFC 9309 says about robots.txt

robots.txt ran for almost thirty years on an informal 1994 convention. In 2022 it was written down as RFC 9309, the Robots Exclusion Protocol, which is the version to implement. Its rules, in brief:

Question	What RFC 9309 says
Which group applies to me?	Match your product token against User-agent lines, case-insensitively. If no group matches, use the * group. If there's no * group either, nothing is disallowed.
Which rule wins?	The most specific match — the rule with the most octets. When an Allow and a Disallow are equally long, Allow should win.
Wildcards?	* matches any sequence of characters; $ anchors the end of the path.
robots.txt returns 4xx	The file is unavailable: you may fetch anything.
robots.txt returns 5xx or can't be reached	Assume complete disallow. After a long outage (the RFC's example is 30 days) you may treat it as unavailable or use a cached copy.
How long can I cache it?	Don't use a cached copy for more than 24 hours, unless the file is unreachable.
Redirects?	Follow at least five, even across hosts; the rules apply to the original host.
How big a file must I parse?	At least 500 KiB.

Two things that aren't in the RFC trip people up. Crawl-delay is a non-standard directive: some clients honour it, Google ignores it, and it costs you little to respect when present. And robots.txt is per scheme and host — https://shop.example.com/robots.txt governs that host alone, not www.example.com.

Checking a URL with Python's standard library

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
rp.can_fetch("MyFetcher", "https://example.com/private/report")  # False if disallowed
rp.crawl_delay("MyFetcher")                                     # None unless the site sets one

Python's parser predates the RFC and differs from it in places — it applies the first matching rule rather than the longest, for one. For anything beyond a script, use a parser that implements RFC 9309, such as Google's open-source robotstxt library or protego in Python.

Pacing

How fast is polite?

There is no standard number, and anyone who gives you one doesn't know the site. A CDN-fronted news site serves a thousand requests a second without noticing; a small shop on shared hosting falls over at five concurrent page loads, because each one costs it several PHP workers plus the assets. The only honest rule is to start gentle and let the host tell you how fast it can go.

1
Limit concurrency per host, not globally. A hundred parallel requests spread over a hundred sites is nothing; a hundred at one site is an incident. Start at one or two in flight per host.
2
Watch response times. Rising latency is the earliest sign of strain — earlier than errors. Slow down when a host's responses get slower.
3
Treat 429 and 503 as "slow down", not "retry now". Honour Retry-After when it's sent. Otherwise back off exponentially, with jitter so your retries don't arrive in waves.
4
Come back down gradually. After a host recovers, speed up one step per success, not straight back to full speed. The first page that loads after an outage proves the server answered once, not that it has recovered.
5
Cache. The politest request is the one you don't send. Re-fetch a page only when you need a newer copy than the one you have.

Backoff that honours Retry-After

import random, time, requests

def polite_get(url, attempts=5, base=2.0):
    for attempt in range(attempts):
        response = requests.get(url, headers={"User-Agent": "MyFetcher/1.0 (+https://example.com/bot)"}, timeout=30)
        if response.status_code not in (429, 503):
            return response
        retry_after = response.headers.get("Retry-After", "")
        wait = int(retry_after) if retry_after.isdigit() else base ** attempt
        time.sleep(wait + random.uniform(0, wait / 2))
    return response

Identify yourself, too: a User-Agent with a name and a URL or email lets a site owner contact you instead of blocking you.

With URLpipe

What URLpipe does for you, and what stays yours

Every URLpipe project follows robots.txt by default. Before a page is fetched, the site's file is read the way RFC 9309 describes — the URLpipe product token's group if there is one, otherwise *; longest match wins, Allow on a tie; each file reused for up to 24 hours. A disallowed page isn't fetched, the request fails with a message that says why, and it costs nothing. One deliberate difference from the RFC: a file that can't be read — a server error, a timeout — is treated like a missing one, and the page is fetched as usual.

A project can turn that off in its settings, for pages you have your own right to fetch — your own sites, a client's under contract. While it's off, having the right to fetch each page and use what comes back is your responsibility, as the Terms set out. Results fetched with it off are stored apart, so a project that follows robots.txt is never served one. The details are in the robots.txt docs.

Pacing is built in. When a target host starts to stall, requests to it are spaced out and the gap grows while the host keeps struggling, then shrinks one step at a time as pages load again; requests wait rather than fail, and you're not billed for the wait. Your own side is bounded by the per-project rate limit and your plan's parallel-request limit — see credits and limits. What stays with you: choosing which pages to fetch, how often, and what you do with them.

FAQ

Frequently asked questions

Is robots.txt legally binding?

robots.txt itself is a convention, not a contract or an access control — RFC 9309 says so explicitly. Whether you may fetch and use a page depends on the site's terms, copyright and the law where you operate. Following robots.txt is the baseline courtesy, not the whole answer.

How often should I re-read a site's robots.txt?

RFC 9309 says crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable. Caching it for up to a day is both polite and allowed.

What if robots.txt returns a 404 or a 500?

Under RFC 9309, a 4xx means no rules — you may fetch. A 5xx or a network failure means you should assume everything is disallowed for a while, because the site may be in trouble.

How many requests per second is polite?

There is no standard number, and the right one depends on the site. Start with one request at a time per host, watch response times, and slow down as soon as they rise or you see 429 or 503 responses. Honour Retry-After when a site sends it.

What is Crawl-delay?

A non-standard robots.txt directive some crawlers honour, asking for a number of seconds between requests. RFC 9309 does not define it, and Google ignores it, but respecting it when present costs you little.

Try it yourself

Free tools for this

No signup — run these on a real page right now, then call the same endpoint from your code.

/robots-txt API docs /credits API docs

Keep reading

Rendered vs. raw HTML Verify webhook signatures (HMAC)Idempotency keys explained Cookie banners and ads in scraped pages HTML to Markdown for LLMs All guides

Put this into practice.

Each of the eight kinds of data URLpipe returns has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 1,000 credits a month, no card.

Start free Email the team