Changelog · September 8, 2026
A site that can't keep up is paced, not failed
A crawl of a small shop on shared hosting should finish, not fail 24 requests in a row.
By Roger Campos · Last updated: September 2026
TL;DR
When a site you're fetching times out, returns a 5xx or says 429, URLpipe now slows its requests to that site and tries again instead of reporting each one as a failure. Requests keep trying for up to 10 minutes from when they were accepted, so a batch against a small site takes longer rather than failing.
Free plan, no credit card. 1,000 credits a month.
A customer's crawl of a small shop failed 24 requests in a row with timeouts. The pages were fine a minute later; the shop's server had simply been overwhelmed. Requests to a struggling site now back off — the gap between requests grows with every stall and shrinks with every page served — and a request that hit a busy moment is retried instead of answered with a failure.
Only the failures worth waiting out are retried: a timeout, the site's own 5xx, and 429. A refused connection, a name that doesn't resolve, a bad certificate or a 404 is answered straight away, because waiting won't change it.
What you see is a slower request, not a failed one. A sync request may outlive its 60-second wait and answer with a token to collect later; an async one simply delivers later. You aren't charged for the wait.
Endpoints
The endpoints on this page
- Get the rendered HTML of any URL
Paste a link and get the page's HTML after JavaScript has run and redirects have been followed — the DOM a real browser sees, not the empty shell curl returns.
Try it free - Turn any URL into clean Markdown
Paste a link and get the page's main content as tidy Markdown — headings, lists, links and code kept, navigation and cookie banners stripped. The format LLMs and RAG pipelines work best with.
Try it free
FAQ
Frequently asked questions
How long will URLpipe keep retrying a slow site?
Which errors are retried?
Am I charged for the waiting?
Try it on the free plan.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.