Caching & freshness

URLpipe caches the result of every operation per URL. When you ask for the same thing again, you can accept the cached result — instantly, and without spending a single credit.

The max_age parameter

Every endpoint accepts an optional max_age parameter that says how fresh a cached result must be for you to accept it. If a cached result exists and is younger than max_age, URLpipe returns it as-is. Otherwise it does the work again and caches the new result.

# Accept a cached result up to 2 hours old
curl -X POST https://urlpipe.dev/markdown \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "max_age": "2 hours"}'

Accepted units & limits

• Seconds: s, sec, second(s)
• Minutes: m, min, minute(s)
• Hours: h, hr, hour(s)
• Days: d, day(s)
• Weeks: w, week(s)

Values are clamped to a range of 0 to 30 days. Set max_age=0 to always bypass the cache and fetch fresh. An unrecognised value returns a 422 with error invalid_max_age.

Cache hits are free

This makes max_age a direct cost lever. Polling a set of URLs every few minutes but happy with hourly freshness? Set max_age="1 hour" and most requests are served from cache for free. Need the absolute latest? Use max_age=0 and every request does real work and spends credits.

A result still on its way counts too: repeat a request while the first is running and the repeat waits for it, for free. See retries & duplicates.

Telling a hit from a miss

Every response says which it was. X-Cache is hit or miss (or partial for a /scrape whose operations differed), and on a hit X-Cache-Age gives the served result's age in seconds — always at or below the max_age you asked for. Together they tell you exactly what a given max_age is buying you before you tune it. See Response headers.

What counts as the same request

Cached results belong to your organization, and every project in it draws on the same cache — one project's fetch is the next project's free hit. They go no further than that: your results answer your requests and nobody else's, and no other account's results are ever handed to you.

Within your organization, a cached result is keyed by the operation, the URL, and any request options that change the output. For /lighthouse, for example, the device and include_audits options are part of the key, so a mobile audit and a desktop audit are cached separately.

residential is part of the key too, and for the same reason: a site that answers a home address differently is the whole point of asking for one, so a residential result is never handed to a request that did not ask for it, and an ordinary result is never handed to one that did. Repeat the same residential call inside your window and it is a free hit like any other.

So is the project's robots.txt setting: a page fetched by a project that does not follow robots.txt is never handed to one that does.