# URLpipe — full documentation > Every page of https://urlpipe.dev/docs, in order. The site overview, pricing and page index is https://urlpipe.dev/llms.txt. --- # URLpipe API URLpipe turns any public URL into clean, structured data. Point it at a page and get back Markdown, metadata, an AI summary, keywords, a Lighthouse audit, a screenshot, raw HTML or the JavaScript console — all through one simple, bearer-authenticated HTTP API. Every endpoint takes the same shape: a `POST` request with a JSON body containing the `url` you want to process. There is nothing to install on the target site — URLpipe renders each page with headless Chrome from our own servers, so JavaScript-heavy and single-page apps work too. ## A first request Send a URL to the [/markdown](https://urlpipe.dev/docs/markdown) endpoint and get back clean Markdown, ready to drop into an LLM prompt or a RAG pipeline. ``` curl -X POST https://urlpipe.dev/markdown \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` [Get set up in the Quickstart](https://urlpipe.dev/docs/quickstart) --- ## The endpoints URLpipe has nine endpoints. Eight return one kind of data each, in two families, and [/scrape](https://urlpipe.dev/docs/scrape) runs any of them together from a single page visit. **Web** endpoints fetch and measure the page directly; **AI** endpoints use a language model to extract or transform its content. One allowance covers both, and each endpoint spends from it what it costs to run — a page fetch is one credit, and the AI endpoints are dearer because a model bills us by the token. See [Credits](https://urlpipe.dev/docs/credits). ### Web endpoints [/htmlWeb The fully rendered HTML of a page, after JavaScript has run.](https://urlpipe.dev/docs/html)[/markdownWeb The page's main content as clean Markdown, chrome stripped.](https://urlpipe.dev/docs/markdown)[/screenshotWeb A full-page screenshot of the rendered page — PNG, JPEG or WebP, base64-encoded.](https://urlpipe.dev/docs/screenshot)[/lighthouseWeb A real Google Lighthouse audit — performance, SEO, accessibility.](https://urlpipe.dev/docs/lighthouse)[/consoleWeb JavaScript console errors, warnings and uncaught exceptions.](https://urlpipe.dev/docs/console) ### AI endpoints [/metaAI Title, description, author, dates, feed and main image.](https://urlpipe.dev/docs/meta)[/summarizeAI A concise AI summary of the page's main content.](https://urlpipe.dev/docs/summarize)[/keywordsAI 5–15 relevant keywords, ordered by relevance.](https://urlpipe.dev/docs/keywords) The same endpoints are also an [MCP server](https://urlpipe.dev/docs/mcp), so an AI agent in Claude, Cursor or any MCP client can call them itself — same arguments, same responses, same credits. --- ## Core concepts [Authentication How bearer tokens and per-project API keys work.](https://urlpipe.dev/docs/authentication)[Caching & freshness Reuse recent results for free with max\_age.](https://urlpipe.dev/docs/caching)[Async & sync modes Webhook the result (the default), verify its signature, or get it inline with sync.](https://urlpipe.dev/docs/async)[Residential exits Fetch a page from a home broadband address, for sites that serve a datacentre less.](https://urlpipe.dev/docs/residential)[Credits & rate limits What each call costs, and why cache hits are free.](https://urlpipe.dev/docs/credits)[Errors Status codes and error payloads to handle.](https://urlpipe.dev/docs/errors) ## Base URL & conventions - All requests go to `https://urlpipe.dev` over HTTPS. - Every endpoint is a POST with a JSON body. - Authenticate with `Authorization: Bearer YOUR_API_KEY` — see [Authentication](https://urlpipe.dev/docs/authentication). - Web endpoints return `text/plain` or `application/json`; failures return JSON with an `error` field. Source: https://urlpipe.dev/docs --- # Quickstart This guide gets you from zero to your first response in a couple of minutes. You'll create an account, grab a project API key, and send your first request. URLpipe is free to start — no credit card. The Free plan includes a monthly allowance of enough credits for both the browser and the AI endpoints, so you can build and test straight away. ## 1. Create an account and a project Sign up, then create your first **project**. Every project has its own API key and its own request history, so it's normal to have one project per app or environment (say, `production` and `staging`). The project's API key is shown to you as soon as it is created — copy it then, because it is shown only once. Confirm your email address and the key is live. [Create your free account](https://urlpipe.dev/auth/sign-up/new) ## 2. Keep your API key safe Treat the key like a password: it authenticates every request. We store only a hash of it, so if you lose it, rotate it from the project's **Settings → API key** and you are shown the new one. See [Authentication](https://urlpipe.dev/docs/authentication) for the details. ## 3. Make your first request Send a `POST` to any endpoint with your bearer token and a JSON body containing the `url`. Here we convert a page to Markdown, with `sync: true` so the result comes straight back in the response: ``` curl -X POST https://urlpipe.dev/markdown \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "sync": true}' ``` That's it — the response body is the Markdown for the page. Every other endpoint, like [/screenshot](https://urlpipe.dev/docs/screenshot) or [/meta](https://urlpipe.dev/docs/meta), takes the exact same request shape. ## 4. Get the result back: async or sync By default, requests are **async**: leave out `sync` and you get a token back immediately, and the result reaches you either at a `report_to` webhook URL or from [GET /result/:token](https://urlpipe.dev/docs/results) whenever you ask for it. That is the better fit for slow work like Lighthouse audits and for anything you run in bulk; `sync: true` suits a quick one-off like the one above. Both are covered in [Async & sync modes](https://urlpipe.dev/docs/async). ## What's next - [Understand authentication and API keys](https://urlpipe.dev/docs/authentication) - [Serve repeat requests for free with caching](https://urlpipe.dev/docs/caching) - [Learn what each call costs in credits](https://urlpipe.dev/docs/credits) - [Browse the full endpoint reference](https://urlpipe.dev/docs/html) Source: https://urlpipe.dev/docs/quickstart --- # Authentication URLpipe authenticates every request with an HTTP bearer token. The token is your project's API key — there is nothing else to configure: no OAuth flow, no signed requests, no session to keep alive. ## Sending your API key Add an `Authorization` header with the value `Bearer YOUR_API_KEY` to every request. The scheme keyword `Bearer` is required, followed by a single space and then the key exactly as issued — no quotes, and no extra whitespace. Requests without a valid key are rejected with `401 Unauthorized` before any work is done, so a rejected request never spends credits. ``` curl -X POST https://urlpipe.dev/markdown \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` ## Confirming your email address A key starts working once the email address on your account has been confirmed, or as soon as the organization is on a paid plan. We send the link the moment you sign up; clicking it once brings every key on the account — this one and any you create later — to life. Until then, on the free plan, the API answers `403 Forbidden`: 403 Forbidden ``` { "error": "email_unverified", "message": "Confirm the email address on this account before using the API." } ``` The key itself is valid, so sending it again — or rotating it for a new one — changes nothing. The dashboard and the Playground are not gated: you can sign in and use both while the address is unconfirmed. If the email never arrived, sign in and use **Account Settings → Send a confirmation link** to get another one. ## Testing your key The quickest way to confirm a key works is to send any real request — the `/markdown` call above against a page you control is a good choice. A `200` (or a `422` about the URL itself) means the key authenticated; a `401` means the header never reached us in a form we accept. When you see a `401`, check that the header name is exactly `Authorization`, that the value starts with `Bearer`, and that no proxy or framework is stripping or rewriting the header in transit. ## How keys are scoped Each API key belongs to a single **project**. A project owns its own request history and draws on its organization's monthly credit allowance. Using separate projects per app or environment keeps history clean and lets you rotate one key without disturbing the others. A common setup is one project per environment — `staging` and `production` — so a leaked staging key can be rotated without touching live traffic, and each environment's usage shows up separately in the dashboard. ## Storing and rotating your key Your key is shown once, when you create the project. We store a hash of it and never the key itself, so it cannot be shown to you again — and a copy of our database is not a copy of your credentials. **Settings → API key** shows only its first few characters, which is enough to tell which key a server is running. Lost it? Rotate it from that screen. You are shown the new key once, and the key it replaces keeps authenticating for 24 hours — so you can deploy the new one at your leisure instead of racing a window where live requests fail with `401`. Rotated because the key leaked? Use **Stop accepting old key** once the new one is deployed. That closes the window immediately: a key we still honour is one that whoever took it can still spend. ## Keeping your key secure Your API key is a secret: anyone who has it can spend your credits and read whatever your account can. Treat it like a password. - Send every request over HTTPS — never plain HTTP, which exposes the key in transit. - Keep the key server-side. Never embed it in browser JavaScript, mobile apps, or any code shipped to users, where it can be extracted. - Load it from an environment variable or a secrets manager. Don't hard-code it in source or commit it to version control — scanners harvest public repositories within minutes. - Use a distinct key per environment so you can rotate one without affecting the others. - Rotate immediately if a key is ever exposed — then stop accepting the old one, rather than letting its 24-hour window run out on its own. ## Authentication errors Two things cause a 401: - The Authorization header is missing. - The token doesn't match any active project key. Everything else — a bad URL, a page that's too big, a timeout — is a `422` with a descriptive body. See [Errors](https://urlpipe.dev/docs/errors) for the full list. Source: https://urlpipe.dev/docs/authentication --- # MCP server URLpipe speaks the Model Context Protocol, so an AI agent can fetch and read pages directly instead of you writing the HTTP calls. Point any MCP client at one endpoint and it can turn a URL into Markdown, a screenshot, metadata or a Lighthouse audit on its own. The tools are the endpoints documented in this site — same arguments, same responses, same credits. There is nothing new to learn: if you know `POST /markdown`, you know `fetch_markdown`. ## Create a token MCP uses its own credential: an organization token, created in [MCP access](https://urlpipe.dev/app/organization/api_tokens) under your organization. It is not the project API key the HTTP endpoints take — a project key names one project, and an agent works across all of them. You choose what a token may do when you create it: - **Fetch pages and read results** — everything. This is what an agent normally needs. - **Read past results only** — can list projects, read usage and retrieve results somebody already paid for, but cannot fetch a page. A token that physically cannot spend your credits. A token is shown once and stored as a hash, so copy it when you create it. Deleting it revokes it immediately. You can also narrow a token to a single project, or give it an expiry date. ## Connect a client This is a remote server, so there is nothing to install and no command to run. A client needs three things, and they are the whole of the setup: - **Transport** — streamable HTTP. The server answers each message with a single JSON response; it opens no SSE stream and keeps no session. - **URL** — `https://urlpipe.dev/mcp` - **Authentication** — an `Authorization: Bearer YOUR_TOKEN` header, the same one every URLpipe request uses. There is no OAuth flow to complete: set the header and the client is connected. How you register a server differs by client — some take these values in a form, some in a command, and most keep them in a JSON file of this shape: MCP client configuration ``` { "mcpServers": { "urlpipe": { "type": "http", "url": "https://urlpipe.dev/mcp", "headers": { "Authorization": "Bearer YOUR_TOKEN" } } } } ``` Keep the token out of that file if the file is in version control. Many clients expand environment variables inside it — check yours. ## The tools 9 tools are the API endpoints. Each takes the same arguments its endpoint does, and answers with the same body. | Tool | Endpoint | Credits | | --- | --- | --- | | fetch\_markdown | [POST /markdown](https://urlpipe.dev/docs/markdown) | 1 | | fetch\_html | [POST /html](https://urlpipe.dev/docs/html) | 1 | | capture\_screenshot | [POST /screenshot](https://urlpipe.dev/docs/screenshot) | 1 | | console\_logs | [POST /console](https://urlpipe.dev/docs/console) | 1 | | lighthouse\_audit | [POST /lighthouse](https://urlpipe.dev/docs/lighthouse) | 2 | | extract\_metadata | [POST /meta](https://urlpipe.dev/docs/meta) | 5 | | extract\_keywords | [POST /keywords](https://urlpipe.dev/docs/keywords) | 15 | | summarize\_page | [POST /summarize](https://urlpipe.dev/docs/summarize) | 17 | | scrape\_url | [POST /scrape](https://urlpipe.dev/docs/scrape) | per operation | 5 more exist only here, because an organization token reaches more than one project and the HTTP API's credential does not: | Tool | What it does | | --- | --- | | list\_projects | Your projects and their ids. Every other tool needs one. | | get\_usage | Credits used and remaining, when they reset, and what each operation costs. | | get\_result | The result of an earlier call, by its token. Free. | | list\_requests | What a project has already fetched, newest first, with each token. Filters by labels. | | get\_request | One request in detail: timings, options, outcome, webhook delivery. | ### Arguments Every endpoint tool takes `project_id` and `url`, plus the optional arguments its endpoint accepts. - **Name** : `project_id` **Type** : integer **Required** : Required **Description** : Which project the request belongs to — it is recorded and billed against that project. Get it from `list_projects`. This is the one argument the HTTP endpoints don't have. - **Name** : `url` **Type** : string **Required** : Required **Description** : The page to fetch. A public http(s) URL. - **Name** : `sync` **Type** : boolean **Description** : Wait for the result and return it. Defaults to `false`, exactly as the API does — see [async & sync modes](https://urlpipe.dev/docs/async). - **Name** : `max_age` **Type** : string **Description** : How fresh a stored result must be to be reused. Defaults to 7 days — see [caching & freshness](https://urlpipe.dev/docs/caching). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a home-ISP address instead of a datacentre one. See [residential exits](https://urlpipe.dev/docs/residential). - **Name** : `report_to` **Type** : string **Description** : Async only: where to deliver the result. Defaults to the project's configured endpoint. Webhooks behave exactly as they do over HTTP. - **Name** : `labels` **Type** : object **Description** : Your own keys to find the request by later, e.g. `{"client": "acme"}`. They come back in the metadata and in `list_requests`, which filters by them. See [labels](https://urlpipe.dev/docs/labels). - **Name** : `idempotency_key` **Type** : string **Description** : Makes the call safe to retry: the same key within 24 hours returns the first call's token and result instead of new work. The `Idempotency-Key` header, as an argument — see [retries & duplicates](https://urlpipe.dev/docs/retries#idempotency-key). ## Async is still the default As with the HTTP API, a call returns a token straight away and the work continues in the background. Pass `sync: true` when you want the agent to wait and be handed the result; otherwise it collects the result with `get_result`. An async call also delivers to the project's webhook if one is configured, and a sync call does not — the same rule, and the same reason, as over HTTP: a sync caller already has the answer in hand. See [async & sync modes](https://urlpipe.dev/docs/async). ## What it costs An MCP call spends exactly what the matching endpoint spends, from the same monthly allowance — there is no separate meter and no surcharge for using an agent. Reused results are free, and a failed fetch is never billed. The [credits page](https://urlpipe.dev/docs/credits) has the full table; `get_usage` returns your live numbers and the same prices, so an agent can check what a run will cost before it starts. Worth telling an agent once: `fetch_markdown` costs 1 credit and `summarize_page` costs 17. If the agent is going to read the page itself anyway, Markdown is both cheaper and the better input — reach for the AI operations when the summary or the keyword list is the thing you are producing. ## Errors and limits A refused call comes back as a tool error whose body is the API's own error object — the same `error` code and message an HTTP caller gets, so `quota_exceeded`, `concurrency_limit` and `invalid_url` mean what [the errors page](https://urlpipe.dev/docs/errors) says they mean. The endpoint is rate limited per token, at a ceiling comfortably above one call per second sustained. Your plan's parallel-request limit applies here too. Source: https://urlpipe.dev/docs/mcp --- # Caching & freshness URLpipe caches the result of every operation per URL. When you ask for the same thing again, you can accept the cached result — instantly, and without spending a single credit. ## The max\_age parameter Every endpoint accepts an optional `max_age` parameter that says how fresh a cached result must be for you to accept it. If a cached result exists and is younger than `max_age`, URLpipe returns it as-is. Otherwise it does the work again and caches the new result. - **Name** : `max_age` **Type** : string | integer **Description** : How old a cached result may be and still be accepted. Accepts a bare number of seconds (`3600`), or a human duration like `"2 hours"`, `"3 days"`, `"30m"` or `"45 sec"`. Defaults to `7 days`. POST /markdown ``` # Accept a cached result up to 2 hours old curl -X POST https://urlpipe.dev/markdown \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "max_age": "2 hours"}' ``` ## Accepted units & limits - Seconds: `s`, `sec`, `second(s)` - Minutes: `m`, `min`, `minute(s)` - Hours: `h`, `hr`, `hour(s)` - Days: `d`, `day(s)` - Weeks: `w`, `week(s)` Values are clamped to a range of `0` to `30 days`. Set `max_age=0` to always bypass the cache and fetch fresh. An unrecognised value returns a `422` with error `invalid_max_age`. ## Cache hits are free A cache hit runs no analysis, so it **neither spends credits nor is blocked by your allowance**. Even if you've hit your monthly limit, cached results keep flowing. This makes `max_age` a direct cost lever. Polling a set of URLs every few minutes but happy with hourly freshness? Set `max_age="1 hour"` and most requests are served from cache for free. Need the absolute latest? Use `max_age=0` and every request does real work and spends credits. A result still on its way counts too: repeat a request while the first is running and the repeat waits for it, for free. See [retries & duplicates](https://urlpipe.dev/docs/retries). ## Telling a hit from a miss Every response says which it was. `X-Cache` is `hit` or `miss` (or `partial` for a [/scrape](https://urlpipe.dev/docs/scrape) whose operations differed), and on a hit `X-Cache-Age` gives the served result's age in seconds — always at or below the `max_age` you asked for. Together they tell you exactly what a given `max_age` is buying you before you tune it. See [Response headers](https://urlpipe.dev/docs/response-headers). ## What counts as the same request Cached results belong to your **organization**, and every project in it draws on the same cache — one project's fetch is the next project's free hit. They go no further than that: your results answer your requests and nobody else's, and no other account's results are ever handed to you. Within your organization, a cached result is keyed by the **operation**, the **URL**, and any **request options** that change the output. For [/lighthouse](https://urlpipe.dev/docs/lighthouse), for example, the `device` and `include_audits` options are part of the key, so a mobile audit and a desktop audit are cached separately. [residential](https://urlpipe.dev/docs/residential) is part of the key too, and for the same reason: a site that answers a home address differently is the whole point of asking for one, so a residential result is never handed to a request that did not ask for it, and an ordinary result is never handed to one that did. Repeat the same residential call inside your window and it is a free hit like any other. So is the project's [robots.txt setting](https://urlpipe.dev/docs/robots-txt): a page fetched by a project that does not follow robots.txt is never handed to one that does. Source: https://urlpipe.dev/docs/caching --- # Async & sync modes By default, URLpipe processes every request asynchronously — it accepts the request instantly and hands you a token. Collect the result at a webhook we POST to, or fetch it by that token whenever you like. Add sync: true to any request to get the result back inline in the HTTP response instead. ## Choosing a mode Pick the mode per request with the `sync` parameter. Omit it (or send `sync=false`) for **async** — the default, which suits long-running work like [/lighthouse](https://urlpipe.dev/docs/lighthouse) and high-volume batches, since you never hold a connection open. Send `sync=true` for **sync** — simplest for quick, interactive calls where you want the answer right away. ## Typical response times How long a request takes depends mostly on the target page — its weight, how much JavaScript it runs, and how fast its own server responds. The figures below are drawn from production traffic for non-cached requests: **p50** is the median and **p90** is the 90th percentile, so 9 in 10 requests finish at or under it. Endpoint Type p50 p90 [`/html`](https://urlpipe.dev/docs/html) Web \3.5 s \4.7 s [`/markdown`](https://urlpipe.dev/docs/markdown) Web \3.5 s \4.7 s [`/screenshot`](https://urlpipe.dev/docs/screenshot) Web \4.0 s \7.6 s [`/keywords`](https://urlpipe.dev/docs/keywords) AI \5.8 s \8.1 s [`/console`](https://urlpipe.dev/docs/console) Web \6.2 s \6.7 s [`/meta`](https://urlpipe.dev/docs/meta) AI \6.3 s \10.8 s [`/summarize`](https://urlpipe.dev/docs/summarize) AI \7.9 s \11.6 s [`/lighthouse`](https://urlpipe.dev/docs/lighthouse) Web \15.3 s \20.5 s Cache hits return in well under a second — they do no work, so tuning [max\_age](https://urlpipe.dev/docs/caching) is the simplest way to make repeat requests fast. AI endpoints add an LLM pass on top of the page fetch, and `/lighthouse` runs a full audit — the slowest by design, and a natural fit for async. ## Async requests Async is the default, so no `sync` parameter is needed. The endpoint responds immediately with a token, and the result reaches you one of two ways: we POST it to your webhook, or you fetch it with [GET /result/:token](https://urlpipe.dev/docs/results) when you are ready. Both are always available — the token in this response is what makes the second one work, whether or not a webhook is configured. That immediate response also carries the usual [metadata headers](https://urlpipe.dev/docs/response-headers): the token, whether the result was already cached (and how old it is), and your remaining quota. The one it can't carry is `X-Processing-Time-Ms` — it measures how long the work took, and on a cache miss the work hasn't started. The number arrives with the result, in the webhook's `meta` object or on the [GET /result/:token](https://urlpipe.dev/docs/results) response. On a cache hit there was nothing to run, so the accept carries it already. ## Where results go Add `report_to` to a request to have that result delivered there. Set a **default endpoint** for the project (Settings → Default Webhook Endpoint) to have every async result delivered without repeating the URL on each call. Both are optional, and the request wins over the project: Request has report\_to Project has a default Where the result goes yes either the request's report\_to no yes the project's default endpoint no no nowhere — fetch it with GET /result/:token A request that names no endpoint and belongs to a project with no default is still **accepted and processed** — it simply produces no webhook. That is the right shape if you poll, or if you queue tokens and collect results in your own time. Naming an endpoint we cannot deliver to, on the other hand, is a `422`: it is a mistake worth reporting, whereas naming none is a choice. ## Which endpoints we accept A webhook endpoint is held to the **same rules as the URL you ask us to analyse** — both are requests we make from our own infrastructure. It must be a public `http`/`https` address on the scheme's default port, at a real domain name. - No IP addresses, in any notation — including decimal, octal, hex, short forms and IPv6 - No `localhost`, bare hostnames, or internal suffixes like `.local` and `.internal` - No credentials in the URL — we would store them and show them back to you - No custom ports: `https://hooks.example.com:8443/x` is refused, `https://hooks.example.com/x` is fine This applies to the `report_to` parameter and to the project default alike, and it is enforced again at delivery time — so an endpoint that stops qualifying is refused rather than called. A refused delivery is recorded on the **Webhooks** page with the reason, and the result stays available from [GET /result/:token](https://urlpipe.dev/docs/results). Developing against a local endpoint? Point `report_to` at a public tunnel (ngrok, Cloudflare Tunnel and the like), or skip the webhook entirely and poll `GET /result/:token` — which needs no endpoint at all. Changing the project default affects requests made after the change. Every async request resolves its destination when we accept it and records it, so a delivery already in flight is never retargeted — and the **Webhooks** page in your dashboard shows exactly where each result was sent, with every attempt. ### Request attributes - **Name** : `url` **Type** : string **Required** : Required **Description** : The page to process. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Set to `true` to process the request synchronously instead. Defaults to `false` (async). Does not affect the [cached result](https://urlpipe.dev/docs/caching) a request maps to. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. POST /markdown ``` curl -X POST https://urlpipe.dev/markdown \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "report_to": "https://your-app.com/webhooks/urlpipe" }' ``` Immediate response ``` HTTP/1.1 200 OK Content-Type: application/json; charset=utf-8 X-Result-Token: 0Zx3…9aQ X-Cache: miss X-Labels: {"client":"acme"} X-Quota-Cost: 1 X-Quota-Limit: 1000 X-Quota-Remaining: 943 X-Quota-Reset: 2026-08-31T23:59:59Z { "token": "0Zx3…9aQ", "status": "accepted", "labels": {"client": "acme"} } ``` ## The webhook delivery When processing finishes, URLpipe sends a `POST` with a JSON body to the endpoint the request [resolved to](https://urlpipe.dev/docs/async#where-results-go). The `token` matches the one from the immediate response, so you can pair it with your original request. A request with no endpoint skips this entirely; the same fields come back from [GET /result/:token](https://urlpipe.dev/docs/results) instead. ### Payload fields - **Name** : `token` **Type** : string **Description** : Correlates with the token returned when you made the request. - **Name** : `operation` **Type** : string **Description** : The endpoint that ran, e.g. `markdown`. - **Name** : `labels` **Type** : object **Description** : The [labels](https://urlpipe.dev/docs/labels) you sent with the request, so you can route the delivery to the client or project it belongs to. `{}` when it had none. - **Name** : `success` **Type** : boolean **Description** : Whether the operation succeeded. - **Name** : `result` **Type** : string | object | null **Description** : The result on success (a string or JSON value, depending on the endpoint); `null` on failure. - **Name** : `result_url` **Type** : string | null **Description** : For [screenshots](https://urlpipe.dev/docs/screenshot), a link to the PNG that needs no API key — the async counterpart of the `X-Result-Url` header. `null` for every other endpoint, and on failure. - **Name** : `error` **Type** : string | null **Description** : A human-readable message on failure; `null` on success. - **Name** : `meta` **Type** : object **Description** : The same metadata a synchronous response returns in its [X- headers](https://urlpipe.dev/docs/response-headers) — an async accept cannot carry facts about work it hasn't done, so they ride here instead, with the result. Present on success and failure alike. Keys: `cache`, `cache_age`, `processing_time_ms` and `quota` (`cost`, `limit`, `remaining`, `overage`, `resets_at`, `concurrency_limit`). A value we don't have is `null` rather than absent. On success ``` { "token": "0Zx3…9aQ", "operation": "markdown", "labels": {"client": "acme"}, "success": true, "result": "# Example Domain\n\nThis domain is for use…", "result_url": null, "error": null, "meta": { "cache": "miss", "cache_age": null, "processing_time_ms": 4182, "quota": { "cost": 1, "limit": 1000, "remaining": 943, "overage": 0, "resets_at": "2026-08-31T23:59:59Z", "concurrency_limit": 3 } } } ``` On failure ``` { "token": "0Zx3…9aQ", "operation": "markdown", "labels": {"client": "acme"}, "success": false, "result": null, "result_url": null, "error": "The request timed out.", "meta": { "cache": "miss", "cache_age": null, "processing_time_ms": 60104, "quota": { "cost": 0, "limit": 1000, "remaining": 944, "overage": 0, "resets_at": "2026-08-31T23:59:59Z", "concurrency_limit": 3 } } } ``` ## Signing & verifying webhooks Your `report_to` URL has to accept an unauthenticated POST from the internet. Turn on **webhook signing** and every delivery carries an HMAC signature instead, so your endpoint can prove the request came from URLpipe and not from someone who learned the URL. It is off by default and enabled per project under **Settings → Webhook Signing** in your dashboard, which is also where the signing secret appears once it is on (it starts with `whsec_`). One secret per project, used for every endpoint's deliveries. ### Headers on a signed delivery - **Name** : `X-URLpipe-Timestamp` **Type** : string **Description** : When we signed this attempt, as Unix seconds. It is part of what is signed, so it cannot be altered — reject anything older than your tolerance (5 minutes is a good default) to bound replays. - **Name** : `X-URLpipe-Signature` **Type** : string **Description** : One or more signatures, comma-separated, each prefixed with its scheme version: `v1=,v1=`. **Split the header and accept the delivery if any value matches** — during a [secret rotation](https://urlpipe.dev/docs/async#rotation) we sign with both the new and the old secret, and matching on the prefix rather than the whole string is what keeps a future `v2` from breaking your handler. ### How it's computed HMAC-SHA256 over the timestamp, a literal `.`, and the raw request body, keyed with your secret. Delivery headers ``` POST /webhooks/urlpipe HTTP/1.1 Content-Type: application/json X-URLpipe-Timestamp: 1756568400 X-URLpipe-Signature: v1=3a1f9c…7b2e ``` Signed string ``` signed = "1756568400" + "." + raw_body signature = HMAC_SHA256(whsec_…, signed) ``` Verify against the **raw request body** — the exact bytes we sent. Parsing the JSON and re-serializing it changes key order and whitespace, and the digest with it. In Rails that is `request.raw_post`; in Express, `express.raw()` rather than `express.json()` on this route. ``` import hashlib, hmac, json, os, time from flask import Flask, request, abort TOLERANCE = 5 * 60 SECRET = os.environ["URLPIPE_WEBHOOK_SECRET"].encode() @app.post("/webhooks/urlpipe") def urlpipe_webhook(): # request.get_data() is the raw body; request.json is not. body = request.get_data() timestamp = request.headers.get("X-URLpipe-Timestamp", "") header = request.headers.get("X-URLpipe-Signature", "") # While rolling verification out, before signing is enabled, # accept a delivery that carries no signature: # if not header: return process(body) if not header or not timestamp.isdigit(): abort(401) if abs(time.time() - int(timestamp)) > TOLERANCE: abort(401) signed = timestamp.encode() + b"." + body expected = "v1=" + hmac.new(SECRET, signed, hashlib.sha256).hexdigest() # Any one of the comma-separated signatures may match: during a # rotation we send the new and the old. if not any(hmac.compare_digest(s.strip(), expected) for s in header.split(",")): abort(401) return process(body) def process(body): enqueue(json.loads(body)["token"]) return "", 200 ``` ## Turning signing on and off Both take effect on the **next delivery**, including a retry of a webhook whose earlier attempts went out the other way. The two directions are not equally safe. **Turning it on is safe at any time.** The signature rides in headers and never changes the payload — same method, same `Content-Type`, byte-identical body — so an endpoint that ignores the headers is unaffected. Enable it first, then add verification whenever you are ready. Doing it the other way round means your endpoint rejects unsigned deliveries before we sign any, so keep the `no signature` branch in the examples below until signing is on. **Turning it off is the breaking direction.** Deliveries stop carrying the headers at once, so an endpoint that requires a signature will reject every result. Remove or relax your verification before you disable it. ## Rotating the secret Rotating in Settings opens a **24-hour grace window** rather than cutting over. For those 24 hours every delivery carries two signatures — the new secret first, then the one it replaced — so an endpoint still holding the old secret keeps verifying while you deploy the new one, in either order and with nothing in flight rejected. After the window, only the new secret is sent. That window is not what you want for a **leaked** secret: signing with the old key is exactly what keeps a leaked key useful to a forger, for as long as your endpoint honours it. Rotate, deploy the new secret, then use **Stop accepting old secret** in Settings to end the window immediately. ## Delivery guarantees - A result that [resolved to an endpoint](https://urlpipe.dev/docs/async#where-results-go) is delivered there as a JSON POST. One that resolved to none is not delivered at all, and waits for you at GET /result/:token - Webhook delivery uses a 30-second timeout. - If delivery fails, the result is still stored and remains visible in your project's history in the dashboard, so nothing is lost. - Each request gets a unique token, so concurrent jobs never collide. - With [signing](https://urlpipe.dev/docs/async#signature) enabled, every delivery — and every retry of one — is signed with your project's secret, carrying a fresh timestamp, so a retry passes a freshness check like any other delivery. With it off, no signature headers are sent. - A failed delivery can be sent again by hand from the **Webhooks** page in your dashboard, at any time — there is no window on it, which matters because the automatic attempts are long finished by the time an endpoint is fixed. - A delivery your endpoint rejects — a failed signature check included — is retried with backoff, up to six attempts in all spread over roughly twenty minutes. After that we stop and email your organization's admins. The result is not lost either way: it stays in your history and behind [GET /result/:token](https://urlpipe.dev/docs/results) for the full retention window. - The `meta` object is **additive** — every other payload field keeps its name, type and meaning, so existing consumers need no changes. - The figures in `meta.quota` are taken when the webhook is delivered, not when the analysis finished, so a retried delivery reports your allowance as of that attempt. - Prefer to pull instead of receive a webhook? Fetch the result any time with [GET /result/:token](https://urlpipe.dev/docs/results). Make your webhook endpoint idempotent and key it on `token` — treat a repeated token as the same result rather than a new one. ## Sync mode Send `sync=true` to process a request synchronously. Webhooks play no part: `report_to` and the project's default endpoint are both ignored, because the response IS the delivery. The request is processed inline and the result comes straight back in the HTTP response body, in the content type documented for each endpoint. Failures return a `422` with an `error` field. Every response — sync or async — carries an `X-Result-Token` header identifying the request, so you can re-fetch the result later via [GET /result/:token](https://urlpipe.dev/docs/results). It arrives alongside the cache status, the processing time and your remaining quota: see [Response headers](https://urlpipe.dev/docs/response-headers). ## Sync timeouts A sync request waits up to **60 seconds** for the analysis to finish. If it isn't ready in time, the endpoint returns `504 Gateway Timeout` with the `token` in the body — but the analysis keeps running in the background. Retrieve the result once it completes via [GET /result/:token](https://urlpipe.dev/docs/results), or use async mode (omit `sync`) for consistently long-running work. 504 Gateway Timeout ``` { "error": "processing_timeout", "message": "The analysis is taking longer than expected. Retrieve it later via GET /result/:token, or use async mode for long operations.", "token": "0Zx3…9aQ" } ``` A `504` comes back with the analysis still running, so it carries no `X-Processing-Time-Ms` header — there is no finished work to report yet. An async accept is the same, unless the result was already cached: then the lookup was the whole of the work and the accept carries the number already. Otherwise it arrives with the result — in the webhook's `meta` object, or on the [GET /result/:token](https://urlpipe.dev/docs/results) response that collects it. Source: https://urlpipe.dev/docs/async --- # Retries & duplicates Retry any request without paying for it twice. An identical request that arrives while the first is still running shares its work, and an Idempotency-Key turns a retry into the first request's answer. ## Identical requests share one run Send the same request again while the first is still working — a client that gave up waiting, a job that ran twice — and the second one waits for the first instead of starting over. It is **free**, and it does not use one of your [concurrent requests](https://urlpipe.dev/docs/credits#concurrency). Each request still gets its own `token`, [labels](https://urlpipe.dev/docs/labels) and [webhook](https://urlpipe.dev/docs/async#webhook), and each is answered with the same result — or, if the page could not be fetched, the same error, which is free for both. Its `X-Cache` is `hit` and its `X-Quota-Cost` is `0`, exactly as for a result served from the [cache](https://urlpipe.dev/docs/caching). Identical means what it means for the cache: the same operation, URL and options, from any project in your organization. A [/scrape](https://urlpipe.dev/docs/scrape) shares a run with another /scrape of the same operations and options. This happens on its own; there is nothing to turn on. A request with `max_age=0` always does its own work. It asks for a page fetched after you asked, and a request already running may have fetched its page before. ## Idempotency-Key When your connection drops before our response arrives, you cannot tell whether we received the request. Send an `Idempotency-Key` header, and resending it with the same key is safe: within **24 hours** it returns the first request instead of starting a new one — the same `token`, the same result, one charge and one webhook. POST /markdown ``` # Generate one key per request, and reuse it only to retry that request curl -X POST https://urlpipe.dev/markdown \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Idempotency-Key: 5f0c8a52-6b1e-4c3d-9a7e-2d4f1b8e9c10" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` - **Name** : `Idempotency-Key` **Type** : header **Description** : Any string of up to 255 printable ASCII characters, with no spaces. A UUID is ideal. Keys belong to the project, so two projects never share one. Over the [MCP server](https://urlpipe.dev/docs/mcp), pass it as the `idempotency_key` argument. A retry is answered in the mode it asks for: an async retry gets the token back, and a `sync=true` retry waits for the result — even when the first request was async. Replayed responses carry `Idempotent-Replayed: true`. ## A key is not a cache `max_age` decides how fresh a result you will accept; the key decides whether this is a request you already sent. So a retry returns the first request's result even with `max_age=0`: you asked for one fresh fetch, and a retry is still that one. For new work, send a new key. ## When the key does not match A key is bound to the request it first arrived with — the operation, URL, options, `max_age`, `report_to` and labels. Reusing it for a different request is refused with a `422`, and nothing runs: 422 Unprocessable Entity ``` { "error": "idempotency_key_reused", "message": "This Idempotency-Key was sent with a different request in the last 24 hours. Use a new key for a new request." } ``` A key outside the rules above is a `422` with error `invalid_idempotency_key`. A request refused for [credits or concurrency](https://urlpipe.dev/docs/credits) never claims its key, so retrying it later with the same key is a new attempt. [See how async results reach you](https://urlpipe.dev/docs/async) Source: https://urlpipe.dev/docs/retries --- # Retrieving results Every request is assigned a token. Use it to fetch the result later with a single GET — handy for async jobs, and for sync requests that timed out before the work finished. ## When to use it - In [async mode](https://urlpipe.dev/docs/async), as a pull alternative to (or backup for) the webhook — poll until the result is ready. An async request needs no `report_to` at all if this is how you collect results. - After a sync request times out. A slow sync call returns `504` with the `token`, and the analysis keeps running in the background — retrieve it here once it's done. GET/result/:token ## Fetch a result ### Path parameter - **Name** : `token` **Type** : string **Required** : Required **Description** : The token identifying the original request. Requests are scoped to your project, so a token from another project returns `404`. ### Where the token comes from - Every response carries it in the `X-Result-Token` header — sync or async, cache hit or miss. See [Response headers](https://urlpipe.dev/docs/response-headers). - Async requests also repeat it in the response body (`token`), as does the body of a `504` timeout. ``` curl https://urlpipe.dev/result/YOUR_TOKEN \ -H "Authorization: Bearer YOUR_API_KEY" ``` ## Responses The status code tells you whether the result is ready, still running, failed, or expired. On `200` the body is exactly what the original endpoint returns. Every one of these responses except the `404` also carries the usual [metadata headers](https://urlpipe.dev/docs/response-headers) — the cache verdict, the processing time and your remaining quota — describing the original request, not this retrieval. Status When Body `200 OK` The result is ready. The result itself, in the operation's format (text or JSON) — identical to the sync response. `202 Accepted` Still being generated. { "status": "processing", "token": "…", "labels": {} } — poll again shortly. `422 Unprocessable Entity` The analysis ran but failed. { "error": "" } — the same failure you'd get synchronously. `404 Not Found` No result for this token under your project. { "error": "not\_found" } `410 Gone` The result is older than the 30-day window. { "error": "stale" } — run a new analysis. `401 Unauthorized` Missing or invalid API key. — Poll on `202`: wait a second or two between calls. Most analyses finish within seconds — see typical times on [Async & sync modes](https://urlpipe.dev/docs/async). ## Retention & staleness Results are retrievable for up to **30 days** — the maximum [cache](https://urlpipe.dev/docs/caching) window. After that, `GET /result/:token` returns `410 Gone` with `{ "error": "stale" }`, and you must run a new analysis to get a fresh result. 410 Gone ``` { "error": "stale", "message": "This result is older than the 30-day retention window. Run a new analysis." } ``` Source: https://urlpipe.dev/docs/results --- # Page options Wait for the page you actually want, and take out what you don't — ads, cookie banners, a newsletter pop-up — before anything is read off it. What goes is gone from every result: the HTML, the Markdown, the metadata, the summary, the keywords, the console and the screenshot. Send them as one `page_options` object on any endpoint that loads the page — every one but [/lighthouse](https://urlpipe.dev/docs/lighthouse). They are included in the operation's usual [credits](https://urlpipe.dev/docs/credits). POST /markdown ``` { "url": "https://example.com/blog/launch", "page_options": { "block_ads": true, "block_cookie_banners": true, "remove_selectors": [".newsletter-signup", "#related"] } } ``` ## Options - **Name** : `wait_for_selector` **Type** : string **Description** : Wait for an element matching this CSS selector before reading the page, up to 10 seconds — for a single-page app whose content arrives after the page has loaded. An element that never appears fails the request, and costs nothing. - **Name** : `delay` **Type** : integer **Description** : Milliseconds to wait after the page has settled, `0`–`10000`. For content, or console errors, that arrive on a timer. - **Name** : `block_ads` **Type** : boolean **Description** : Block the major ad networks' requests before the page loads, and remove the slots they would have filled. Pages with ads usually load much faster with it. - **Name** : `block_cookie_banners` **Type** : boolean **Description** : Remove the cookie-consent banners of the platforms that serve most of the web's — OneTrust, Cookiebot, Usercentrics, Didomi, Quantcast, Sourcepoint and more — including the cookie policy text their preference panels put in the page. Nothing is clicked, so no consent is given on your behalf. - **Name** : `remove_selectors` **Type** : array **Description** : CSS selectors of your own elements to remove — a chat widget, a promo bar, a related-posts list. Up to 50. ## Removed, not hidden Everything these options take out is removed from the page, not hidden in it, so it is absent from the HTML and from everything derived from it. On onetrust.com, `block_cookie_banners` takes about 16,000 characters of cookie policy out of the page's text; on elmundo.es, `block_ads` roughly halves the time the page takes to load. To change only how a screenshot *looks* — hide an element in the image but keep it in the HTML — use [screenshot\_options.hide\_selectors](https://urlpipe.dev/docs/screenshot#options) instead. ## Stored results Page options are part of what identifies a result: a page with its ads removed is a different result from the page with them, and each is [stored and reused](https://urlpipe.dev/docs/caching) on its own. An option set to its default is the same request as one that leaves it out. ## Why not /lighthouse A Lighthouse audit loads the page for itself, and measures that load. Waiting on it, blocking its requests or trimming it would change the very scores the audit reports, so `/lighthouse` refuses `page_options` with `invalid_options`. In a [/scrape](https://urlpipe.dev/docs/scrape), they apply to every operation except the audit. Source: https://urlpipe.dev/docs/page-options --- # Labels Tag a request with your own ids — the client, project or campaign it is for — and they come back with its result and its webhook. Your dashboard filters history by them and shows what each one spent, month by month. Any endpoint takes them ``` { "url": "https://example.com", "labels": { "client": "acme", "project": "spring-launch" } } ``` They work on every endpoint, including [/scrape](https://urlpipe.dev/docs/scrape) (one set for the whole request), and over the [MCP server](https://urlpipe.dev/docs/mcp). A form-encoded request sends them as `labels[client]=acme`. ## Where they come back - In the `X-Labels` header of every response for the request — sync results, [GET /result/:token](https://urlpipe.dev/docs/results) and timeouts alike — as a JSON object. See [Response headers](https://urlpipe.dev/docs/response-headers). - In the body of an async accept, beside the `token`: `{"token": "…", "status": "accepted", "labels": {"client": "acme"}}`. - In every [webhook delivery](https://urlpipe.dev/docs/async#webhook), as a top-level `labels` object — route a result to the client it belongs to without keeping a table of tokens. - In the MCP tools' metadata, and in `list_requests`, which filters by them. ## In your dashboard Every project's **History** shows each request's labels. Click one to see every request carrying it, or choose **Filters → Labels** and enter a key and a value. The project's **Labels** page totals requests, failures and credits for each value of a key, one calendar month at a time — the same months, and the same counting, as your [credits](https://urlpipe.dev/docs/credits). It also shows what requests without that label spent, so the figures add up to the project's month and you can bill each client for exactly their share. ## The rules - An object of up to **16 keys**. - Keys are 1 to 40 letters, digits, underscores, hyphens or periods — `client_id`, `project.name`, `env-2`. Case counts: `Client` and `client` are two keys. - Values are strings of 1 to 256 characters, any language. Send ids as strings — `"42"`, not `42` — so what you filter by later is exactly what you sent. - Filters match a value exactly, character for character. Labels outside these rules are refused with a `422` before anything runs or is charged, and the message names what to change: 422 Unprocessable Entity ``` { "error": "invalid_labels", "message": "labels.client must be a string." } ``` ## Free to add Labels cost nothing and change nothing about the work. They are not part of the [cache key](https://urlpipe.dev/docs/caching): a labelled request reuses a stored result exactly as an unlabelled one does, for free, and each request keeps its own labels — two clients fetching the same page each see their own. A request's labels are fixed when you make it. Use ids rather than names, and keep secrets and personal data such as email addresses out of them: they appear in your dashboard, your webhooks and your team's view of the project. [See how async results reach you](https://urlpipe.dev/docs/async) Source: https://urlpipe.dev/docs/labels --- # Residential exits Some sites read the address a request came from and serve a server farm less than they serve a person. Add one parameter and URLpipe fetches the page from a home broadband line instead. Any endpoint takes it ``` { "url": "https://example.com", "residential": true } ``` It works on every endpoint, including [/scrape](https://urlpipe.dev/docs/scrape), and it changes nothing else about the request: the same response, the same headers, the same sync and async modes. It defaults to `false`, so nothing you already run is affected. ## When it helps Try without it first. Most pages come back complete on the standard network, and when a site puts a bot check in front of us we work it and retry before you ever hear about it. Reach for a residential exit when that isn't enough: - The response is `The page asked us to complete a bot check before it would load.` — a site that will not clear for a datacentre address, however long we wait. - The page loads but comes back thin: a paywall stub, a consent wall, or a product page with the prices missing. - You need what a real visitor sees — regional pricing, local inventory, the page a household connection is served rather than the one a bot is. It is not a guarantee, and nothing is. A site that blocks every automated visitor blocks this one too. What changes is the single signal most sites act on first. ## What it costs A residential exit adds **25 credits** to a request — charged once per **page fetch**, not once per operation. Residential bandwidth is bought by the gigabyte and a page visit spends a page's worth of it, which is the whole of why it is priced separately from the work done on the page afterwards. | Request | Operations | Exits | Total | | --- | --- | --- | --- | | `/html` | 1 | 25 | 26 | | `/summarize` | 17 | 25 | 42 | | `/scrape — html, markdown, meta` | 7 | 25 | 32 | | `/scrape — html, lighthouse` | 3 | 50 | 53 | That is the one place a [/scrape](https://urlpipe.dev/docs/scrape) is cheaper than the same operations called one at a time: it visits the page once and reads every operation off that single visit, so it pays for one exit however many results come back. A `/lighthouse` runs its own audit on its own engine, so a scrape that includes it fetches twice and pays twice. The usual rules hold. A result served from [cache](https://urlpipe.dev/docs/caching) fetched nothing, so it costs nothing — exit included. A failed analysis costs nothing either, even though the fetch was attempted. And `X-Quota-Cost` on every response reports the whole figure, exit and operations together. ## Residential results are kept apart The point of asking for a home address is that the page may answer differently, so a residential result is stored under its own key. A residential request is never handed a result fetched from the datacentre, and an ordinary request is never handed a residential one. Repeat the same residential call inside your `max_age` window and it is a free cache hit like any other. ## When no exit is free Residential capacity is finite and, unlike the rest of the network, it has no stand-in: answering from a datacentre address would be the one thing the request asked us not to do. So when nothing is free the request fails, plainly and for free: 422 Unprocessable Entity ``` { "error": "No residential exit was free to load this page. Retry shortly, or send the same request without residential to use our standard network." } ``` Nothing is spent. Retry in a moment, or drop the parameter and take the standard network's answer — which for most pages is the same answer. ## Trying it out Point it at the targets that are giving you trouble and compare. A [free API key](https://urlpipe.dev/auth/sign-up/new) comes with 1,000 credits a month — enough to fetch 38 pages through a residential exit and find out whether it changes what they send back, before you build anything on it. The free [tools](https://urlpipe.dev/tools) and the in-app Playground always use the standard network, so this is the one part of URLpipe you can only try with a key. [See what everything else costs](https://urlpipe.dev/docs/credits) Source: https://urlpipe.dev/docs/residential --- # robots.txt Each project chooses whether its requests follow the robots.txt of the sites they fetch. Following it is the default for every project, and one switch in the project's Settings changes it. ## The setting - **On** — before a page is fetched, we read the site's robots.txt. A page it disallows is not fetched, and the request fails with an error that says why. Nothing is spent. - **Off** — pages are fetched whatever robots.txt says, and the file is never read. Find it under **Settings → robots.txt** in each project. It applies to every endpoint, [/scrape](https://urlpipe.dev/docs/scrape) and the [MCP server](https://urlpipe.dev/docs/mcp) included, and a request keeps the setting it was made under even if the switch changes while it is queued. Turning it off is your call, and so is what follows from it: while it is off, having the right to fetch each page and use what comes back is your responsibility, as the [Terms](https://urlpipe.dev/terms) set out. ## A disallowed page 422 Unprocessable Entity ``` { "error": "The site's robots.txt disallows this page, and this project is set to follow robots.txt." } ``` The same sentence is in the webhook and in [GET /result/:token](https://urlpipe.dev/docs/results) for an async request. The site is never asked for the page, and like every failed analysis the request [costs nothing](https://urlpipe.dev/docs/credits). A `/scrape` fails every operation that would have read the page. ## How the rules are read The file is read the way [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) describes: - The group for `User-agent: URLpipe` applies if there is one; otherwise the `User-agent: *` group does; otherwise nothing is disallowed. - Rules match the page's path and query string. The longest matching rule wins, and an `Allow` beats a `Disallow` of the same length. `*` matches any run of characters and a trailing `$` anchors the end. - Each scheme and host has its own file: `https://shop.example.com/robots.txt` governs that host alone. - A file is reused for up to 24 hours, so a change to it can take that long to apply. Only a rule refuses a page. A site with no robots.txt, or one whose file could not be read — a server error, a timeout — has its pages fetched as usual, and if the site itself is down the request reports that instead. ## Results are kept apart A project that follows robots.txt is never handed a result fetched regardless of it. Results are [shared across your organization's projects](https://urlpipe.dev/docs/caching#cache-key), so a page fetched by a project with the setting off is stored under its own key, and the same request from a project with it on fetches — and checks — afresh. ## For site owners Address URLpipe in your robots.txt by its product token, `URLpipe`: robots.txt ``` User-agent: URLpipe Disallow: /account/ Allow: /account/help ``` We read the file with this user agent: User-Agent ``` Mozilla/5.0 (compatible; URLpipe/1.0; +https://urlpipe.dev/docs/robots-txt) ``` Rules are followed only by projects that have the setting on. The pages themselves are loaded by a standard browser, which does not identify itself as URLpipe. [See every error message](https://urlpipe.dev/docs/errors#failures) Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/robots-txt-checker) Source: https://urlpipe.dev/docs/robots-txt --- # Credits & rate limits Your plan includes a monthly allowance of credits. Every successful call spends some, according to what that call costs us to run — so a plan stretches a long way on cheap operations and less far on expensive ones. ## What each call costs One credit is one page fetch. Everything else is priced relative to that. This table is generated from the same source the API meters with, so it is never out of date: | Endpoint | Credits | Calls a Free plan buys | | --- | --- | --- | | [/console](https://urlpipe.dev/docs/console) | 1 | 1,000 | | [/html](https://urlpipe.dev/docs/html) | 1 | 1,000 | | [/markdown](https://urlpipe.dev/docs/markdown) | 1 | 1,000 | | [/screenshot](https://urlpipe.dev/docs/screenshot) | 1 | 1,000 | | [/lighthouse](https://urlpipe.dev/docs/lighthouse) | 2 | 500 | | [/meta](https://urlpipe.dev/docs/meta) | 5 | 200 | | [/keywords](https://urlpipe.dev/docs/keywords) | 15 | 66 | | [/summarize](https://urlpipe.dev/docs/summarize) | 17 | 58 | One option sits outside this table because it is priced per page fetch rather than per call: [residential exits](https://urlpipe.dev/docs/residential), covered below. The split is not arbitrary. `/html`, `/screenshot` and `/console` all come from one visit to the page in a real headless Chrome and cost us the same; `/lighthouse` runs a full audit that takes minutes of CPU in a limited lane, so it is worth two. `/markdown` converts the page we have already rendered, in the browser and without a model, so it costs a page fetch and nothing more. The three AI endpoints send the page to a language model, and what they pay depends on how much of it the model has to read and write back: `/meta` reads a page's head and answers in a few hundred tokens, while `/summarize` reads the whole article and writes several paragraphs back. ## Fetching from a residential address One option is priced on its own, because what it buys is not work on the page but the connection the page arrives over. [Residential exits](https://urlpipe.dev/docs/residential) fetch a page from a home broadband address instead of a datacentre one, for the sites that answer the two differently. They add **25 credits** per page fetch — once per request, not once per operation, so a `/scrape` pays for one exit however many results it takes off the visit. ## A worked example Say a month's traffic is 500 `/html` calls, 100 `/meta` lookups and 25 `/summarize` calls, and a third of the requests repeat a URL you've already fetched: Working out a month ``` Raw calls 500 /html + 100 /meta + 25 /summarize Cache hits (33%) −165 /html −33 /meta −8 /summarize → free Billable 335 /html 67 /meta 17 /summarize 335 × 1 = 335 credits 67 × 5 = 335 credits 17 × 17 = 289 credits --- 959 credits → fits the Free plan (1,000) ``` The [calculator on the pricing page](https://urlpipe.dev/pricing#calculator) does this for you, including the cache-hit discount. ## Allowance per plan These are the monthly allowances each plan includes. They are read from the same source the API enforces, so this table is never out of date. | Plan | Credits / month | In parallel | Page fetches | /summarize calls | | --- | --- | --- | --- | --- | | Free | 1,000 | 1 | 1,000 | 58 | | Starter | 20,000 | 3 | 20,000 | 1,176 | | Pro | 55,000 | 8 | 55,000 | 3,235 | | Scale | 175,000 | 20 | 175,000 | 10,294 | See [pricing](https://urlpipe.dev/pricing) for what each plan costs, or the Billing screen in your dashboard to change plan. Every paid plan keeps serving past its allowance, uncapped, at $1.50 / 1,000 — see [overage](https://urlpipe.dev/docs/credits#overage). Volumes past Scale run on a contract we size with you — email us. ## Cache hits are free A request served from cache spends nothing and is never blocked by your allowance. Tuning [max\_age](https://urlpipe.dev/docs/caching) is the single most effective way to bring your usage down — it is the one lever that reduces the bill without reducing what your application does. The same goes for a repeat of a request that is still running: it shares the first one's work and spends nothing. See [retries & duplicates](https://urlpipe.dev/docs/retries). ## Failed requests are free too You are only charged for work that produced a result. However an analysis fails — a DNS failure, a refused connection, a timeout, a bad certificate, a page that 404s, or a site that puts a bot check in front of it we cannot clear — the call spends nothing. That includes the residential surcharge on a fetch that came back with nothing to sell you, and it includes a request we could not find a residential exit for at all. ## Reading your balance from a response You don't have to wait for a `429` to find out where you stand. Every response says what it cost and what is left: - **Name** : `X-Quota-Cost` **Type** : integer **Description** : What this request spent. Zero on a cache hit or a failed analysis, and on a `/scrape` the sum over the operations that actually did work — plus any [residential](https://urlpipe.dev/docs/residential) exit it used. On a `429` it is what the request *would* have cost — the figure that didn't fit. - **Name** : `X-Quota-Limit` **Type** : integer **Description** : The monthly credit allowance included in your plan, or `unlimited`. - **Name** : `X-Quota-Remaining` **Type** : integer **Description** : What is left of that allowance after this request. The count is taken after your request, so it's what you have now. - **Name** : `X-Quota-Overage` **Type** : integer **Description** : Credits spent past the allowance this month. Zero until it runs out; after that, what will be billed when the month closes. There is no ceiling on it. - **Name** : `X-Quota-Reset` **Type** : timestamp **Description** : When the allowance rolls over, as an ISO 8601 timestamp. One allowance covers every endpoint, so one pair of numbers describes it however many operations a request ran. The full reference is in [Response headers](https://urlpipe.dev/docs/response-headers). ## Running out never stops you: overage On a paid plan, spending the last of your monthly credits changes the price and nothing else. Your calls keep working — there is no cap, no ceiling and no cut-off — and every credit past the allowance costs **$1.50 / 1,000**. Nothing is charged mid-request: the extra credits are totalled when the month closes and arrive as a single invoice of their own, itemised down to the block. | Plan | Included credits | Each extra 1,000 | 10,000 extra would cost | | --- | --- | --- | --- | | Free | 1,000 | — | Requests are refused | | Starter | 20,000 | $1.50 | $15.00 | | Pro | 55,000 | $1.50 | $15.00 | | Scale | 175,000 | $1.50 | $15.00 | Overage is deliberately dearer than the credits your plan already includes — it is there so a spike is never refused, not so it can replace the right plan. A Starter account settling at Pro-sized traffic pays more in overage than Pro costs, and the billing screen says so while it is happening. Overage is billed in whole blocks of `1,000` credits, and a partial block counts as a full one — 1,200 extra credits is two blocks, $3.00. That keeps a busy month from producing an invoice for four cents. ## Knowing you have crossed over Three ways, none of which require you to go looking. Every response carries `X-Quota-Overage`, the credits you have spent past the allowance this month — zero until you cross, and after that the number that gets billed. We email your organization admins the first time in a month you cross. And the app shows a banner, with the running total, for as long as you are over. There is deliberately no header saying how much further you can go, because there is no answer: on a plan with a card on file you are never cut off for credits. ## When you do run out Only the Free plan is ever refused, and only because there is no card to bill the extra to. When it happens URLpipe returns `429 Too Many Requests` with a JSON body describing exactly where you stand. Nothing is executed and nothing is spent — a `/scrape` that can't afford all of its operations runs none of them, rather than silently dropping some. 429 Too Many Requests ``` { "error": "quota_exceeded", "message": "Your Free plan includes 1000 credits per month. You've used 990 and this request needs 17. Quota resets on 2026-10-01 (UTC).", "limit": 1000, "used": 990, "needed": 17, "resets_at": "2026-10-01T00:00:00Z" } ``` - **Name** : `error` **Type** : string **Description** : Always `quota_exceeded` for this response. - **Name** : `limit` **Type** : integer **Description** : The monthly credit allowance included in your plan. - **Name** : `used` **Type** : integer **Description** : Credits you've spent this period. - **Name** : `needed` **Type** : integer **Description** : What the refused request would have cost. Worth reading: a request for a cheaper operation may still fit. - **Name** : `resets_at` **Type** : timestamp **Description** : When the allowance resets — the 1st of next month, UTC — as an ISO 8601 timestamp. ## Handling it gracefully - Watch `X-Quota-Overage`. On a paid plan no 429 is coming, so this is the only signal there is: the moment it goes above zero you're paying per call, and that is the point at which a bigger plan usually costs less than carrying on. - Read `resets_at` and back off until then rather than retrying immediately. - Widen `max_age` so more requests are served from cache for free. - Reach for a cheaper endpoint where it will do. If you only need a page's title and description, `/meta` costs less than a third of `/summarize`. - Drop `residential` from the calls that do not need it. Most pages come back complete on the standard network, and the exit is the dearest thing on the rate card. - Watch the breakdown on the Billing screen — it itemises the month by operation, residential fetches on a line of their own, so you can see which endpoint is actually spending your allowance. ## Dormant free organizations A free organization that nobody signs in to and that makes no API request for six months is treated as dormant. We email its admins and show a notice in the app; 30 days later the organization and its request history are deleted. Anything counts as activity — one sign-in, one request. So a project whose only job is a nightly cron against `/markdown` never goes dormant, and neither does an account somebody logs into now and then. If a notice does arrive, using the organization cancels it, as does upgrading it or the button in your organization settings. Paid plans are never affected. The full policy is in [the terms](https://urlpipe.dev/terms). ## Parallel requests Separately from how many credits a month buys, each plan sets how many requests you can run **in parallel**. Credits are how much work you can do in a month; this is how fast you get through it. | Plan | Parallel requests | What that suits | | --- | --- | --- | | Free | 1 | Cron jobs, a page per visitor, and steady background work. | | Starter | 3 | A small worker pool, or a busier site fetching pages on demand. | | Pro | 8 | Batch jobs that have to finish inside a window. | | Scale | 20 | Crawls and pipelines, where wall-clock time is the point. | A crawl of 5,000 pages costs the same credits however fast you run it — running 20 in parallel instead of one at a time is the difference between minutes and hours. Every response carries `X-Concurrency-Limit`, so a worker pool can size itself from the plan rather than by trial and error. A request counts from the moment we accept it until it finishes, which for an [async](https://urlpipe.dev/docs/async) request includes the time it spends queued waiting for us to start it. A `/scrape` is one request however many operations it runs, because they come from a single page visit. And a [cache hit](https://urlpipe.dev/docs/caching) occupies no slot at all — it does no work — so widening `max_age` buys you concurrency as well as lowering your bill. Go over and the request is refused with `429 Too Many Requests` and `error: "concurrency_limit"`. Nothing is spent and nothing is queued — retry as soon as one of your own requests finishes. 429 Too Many Requests ``` { "error": "concurrency_limit", "message": "Your Starter plan runs 3 requests in parallel, and 3 are already running. Retry when one finishes, or move to a plan that runs more in parallel. Nothing was spent on this request.", "limit": 3, "running": 3 } ``` There is no `Retry-After` on this response, deliberately: the answer is "when one of yours finishes", not a fixed number of seconds. A client that tracks its own in-flight requests always knows sooner than a header could tell it. Your limit is about your own work. We separately cap how many requests run against any **one target site** at a time, across all customers together, so URLpipe never hammers somebody else's server — and when a site starts struggling, we slow down further until it recovers. Those requests queue rather than being refused, and time a request spends waiting on a site we are pacing does not count against your parallel limit. You didn't cause that wait, so you don't pay for it. ## Rate limits Separately from your monthly allowance, every request is rate limited per project (keyed by your API key) so a runaway script can't flood the service. Two ceilings apply at once: - Up to `60` requests per minute. - Up to `15` requests per 10 seconds (a short-burst ceiling). Exceed either and URLpipe returns `429 Too Many Requests` with a `Retry-After` header giving the seconds to wait. This response is distinct from the credit one above — its `error` is `rate_limited`, so you can tell "slow down" apart from "you've used your allowance". 429 Too Many Requests ``` { "error": "rate_limited", "message": "Too many requests. Slow down and retry after 12 seconds.", "retry_after": 12 } ``` So three different things can answer `429`, and the `error` field is what tells them apart: `rate_limited` means you are sending too fast, `concurrency_limit` means too many of yours are already running, and `quota_exceeded` means the month's credits are gone. Each one tells you what to do next: slow down, wait for one of your own requests to finish, or wait for the 1st. Cache hits count toward the rate limit even though they spend no credits — the limit protects the service from request floods, not just from expensive work. Source: https://urlpipe.dev/docs/credits --- # Data residency Every page you send URLpipe is fetched, rendered and stored on our own servers in Europe, on every plan. Turn on EU processing and the AI model runs there too, so no part of a request leaves the EU. ## What already runs in Europe We run our own infrastructure rather than renting capacity wherever it is cheapest. That means the parts of a request you might expect to be spread across regions are not: - The headless Chrome that loads your URL, and the network it fetches from - The Postgres that holds your account, your request history and your stored results - Every operation that never calls an AI model — `/html`, `/markdown`, `/screenshot`, `/console` and `/lighthouse` So for most of the API the answer is already yes, with nothing to switch on. `/markdown` is worth calling out: it converts the rendered page itself rather than asking a model to, so the cleanest way to feed European web content to your own LLM never leaves Europe either. ## What the EU setting changes Three operations ask an AI model a question: `/meta`, `/keywords` and `/summarize`. That model is the one step that can sit outside Europe. Switch your organization to EU processing and those calls go to a European endpoint instead, where they are decrypted inside the EU and served only by EU providers — with no fallback to a host outside it. Turn it on in your dashboard under **Organization → Data residency**. It applies to every project and every API key in the organization, from your next request on. Organization admins can change it. ## What stays the same - The same endpoints, the same request and response shapes, the same API keys - The same credit costs — EU processing carries no surcharge - The same retention guarantee. On either setting we route only to hosts that neither log your prompts nor keep them, so the model provider retains nothing either way. Residency is the part the setting changes. ## What to expect when you switch - The European model is a different one, so summaries and keyword lists may read a little differently. The response shape does not change. - Results stored before you switched are still served from your history until they age out. Send `max_age=0` on a request to skip stored results and have it processed fresh under the new setting. ## A note on what the law actually asks The GDPR does not require personal data to stay inside the EU — transfers are lawful under an adequacy decision or standard contractual clauses. Residency is usually asked for by a customer, a procurement process or a data protection officer who would rather not run a transfer impact assessment at all. This setting is how you give them that answer without changing anything else about your integration. For what we store, for how long, and who processes it, see the [privacy policy](https://urlpipe.dev/privacy). Source: https://urlpipe.dev/docs/data-residency --- # Response headers Every response carries a set of X- headers describing the request behind it: its token, whether it came from cache and how stale that cache was, how long we took, what it cost you in credits, and what's left of your allowance. They're the metadata channel — the body stays exactly what you asked for. Half the endpoints answer in plain text — [/markdown](https://urlpipe.dev/docs/markdown), [/html](https://urlpipe.dev/docs/html), [/screenshot](https://urlpipe.dev/docs/screenshot) and [/summarize](https://urlpipe.dev/docs/summarize) hand you a body you can pipe straight into a file. Wrapping that in a JSON envelope to carry metadata would break it, so the metadata rides in headers instead. The same headers appear on the JSON endpoints, so one parser reads them everywhere. ## Reference Header Value When it's present `X-Result-Token` string Always. The token identifying this request, for GET /result/:token. `X-Cache` hit | miss | partial Always. Whether the body was served from the result store. `X-Cache-Age` integer (seconds) Only when X-Cache is hit or partial — how old the served result is. `X-Processing-Time-Ms` integer (milliseconds) Only once the work has finished. Absent on a 504, and on an async accept that had work to do. `X-Result-Url` string (URL) Screenshots only. A link to the PNG that needs no API key — put it straight in an tag. `X-Labels` JSON object Only when the request had labels — the labels it was made with, e.g. {"client":"acme"}. ASCII-only: other characters arrive escaped as \\uXXXX, which any JSON parser reads back. `X-Quota-Cost` integer Always. What this request spent, in credits — 0 on a cache hit or a failed analysis, and on a 429 what it would have cost. `X-Quota-Limit` integer | unlimited Always. The monthly credit allowance included in your plan. `X-Quota-Remaining` integer | unlimited Always — what is left of that allowance, counted after this request. `X-Quota-Overage` integer Always. Credits spent beyond the allowance this month — 0 until it runs out, uncapped after. `X-Quota-Reset` ISO 8601 timestamp Always — when the monthly period rolls over. `X-Concurrency-Limit` integer | unlimited Always. How many requests your plan can run in parallel. `Idempotent-Replayed` true Only when the response belongs to an earlier request with the same Idempotency-Key — see Retries & duplicates. A sync /markdown response ``` HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 X-Result-Token: 0Zx3RkP9…9aQ X-Cache: hit X-Cache-Age: 5400 X-Processing-Time-Ms: 12 X-Quota-Cost: 0 X-Quota-Limit: 1000 X-Quota-Remaining: 943 X-Quota-Overage: 0 X-Quota-Reset: 2026-08-31T23:59:59Z X-Concurrency-Limit: 1 # The page, as Markdown … ``` A header that has no honest value is **omitted** rather than sent as an empty string or a zero. Check for presence before parsing. ## On the webhook When an async result is delivered by [webhook](https://urlpipe.dev/docs/async#where-results-go), that POST carries the same facts in its body under a `meta` object — an accept cannot carry facts about work it hasn't done yet, and you never see headers on a request we make to you. It's an **additive** field: every existing key keeps its name, type and meaning. A result you fetch with [GET /result/:token](https://urlpipe.dev/docs/results) instead carries them as headers, as any response of ours does. The meta object ``` "meta": { "cache": "hit", "cache_age": 5400, "processing_time_ms": 12, "quota": { "cost": 0, "limit": 1000, "remaining": 943, "overage": 0, "resets_at": "2026-08-31T23:59:59Z", "concurrency_limit": 1 } } ``` Same values, JSON-typed: integers stay integers, and a fact we don't have is an explicit `null` rather than an absent key — a header consumer checks presence, a webhook consumer parses one shape. The one exception is an unlimited plan, where `limit`, `remaining` and `concurrency_limit` read `"unlimited"` exactly as the headers do, so you branch the same way on either channel. The quota figures are taken when the webhook is **delivered**, not when the analysis finished. If a delivery is retried, its numbers are current as of that attempt. ## Which responses carry them - Sync responses — cache hits and cache misses alike. - Async accepts (`{ "status": "accepted" }`), where they tell you the token, whether the result was already cached — and, when it was, how long that took, since a cache hit is a finished request already. - Every [GET /result/:token](https://urlpipe.dev/docs/results) response, including `202` while it's still running and `410` once it's expired. - Failures with a record behind them: a `422` analysis failure, a `504` sync timeout, and a `429 quota_exceeded` (which carries the quota headers). The exceptions are responses with no request behind them: `401`, a `422` for a malformed `url` or `max_age`, a `404` for an unknown token, and the `429 rate_limited` thrown before the request reaches the endpoint. ## The result token Every request — sync or async, cached or not — gets a token, and `X-Result-Token` is where you read it. Keep it if you might want the result again: fetching it with [GET /result/:token](https://urlpipe.dev/docs/results) is free, doesn't re-run the analysis, and works for the full `30 day` retention window. Async responses also repeat it in the body as `token`. ## Cache status and age A [cache hit](https://urlpipe.dev/docs/caching) is free and instant, so it's worth knowing when you got one. `X-Cache` tells you which happened: - `hit` — served from the result store. No analysis ran, and no credits were spent. - `miss` — the analysis ran for this request (or, on an async accept, is about to). - `partial` — a [/scrape](https://urlpipe.dev/docs/scrape) only: some operations were cached and others weren't. The per-operation `cached` flag in the body says which. When something was served from cache, `X-Cache-Age` gives its age in seconds — the time since that result was computed, not since it was last requested. It's always at or below the `max_age` you asked for, so it tells you how much freshness headroom you actually have. On a `partial` scrape it's the **oldest** operation's age: nothing in the body is staler than that. Halve your bill by checking what you're paying for ``` # Ask for an hour-old result, then see what you got curl -sD - -o /dev/null -X POST https://urlpipe.dev/markdown \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "sync": true, "max_age": "1 hour"}' \ | grep -i '^x-cache' # x-cache: hit # x-cache-age: 2143 ← 36 minutes old, well inside the hour ``` ## Processing time X-Processing-Time-Ms is your **total processing time**: from the moment we accept the request to the moment we finish it, in milliseconds. One number, measured the same way for every request — sync, async with a webhook, async without one. Queueing and analysis are both in it. If a request waited behind others for a free slot against the same site, that wait is yours too and it is in here — there is no separate queue-time or work-time figure to reconcile. What is **not** in it is our webhook delivery. How long our delivery queue took to reach your endpoint, and how many times we had to retry, say nothing about how long your analysis took — so a retried delivery reports the same number as the first attempt, and a request you collect from [GET /result/:token](https://urlpipe.dev/docs/results) reports the same number as one we delivered. It's the number to compare against your own client-side timing — the difference is network. And it's absent until there is something to report: an [async accept](https://urlpipe.dev/docs/async) on a cache miss has no finished work behind it yet, and a `504` means the analysis is still running. It arrives with the finished result, either way: in the [webhook's meta object](https://urlpipe.dev/docs/async), or on the [GET /result/:token](https://urlpipe.dev/docs/results) response that collects it. ## What it cost, and what is left Four numbers, on every response. `X-Quota-Cost` is what this request spent, in [credits](https://urlpipe.dev/docs/credits) — 17 for a [/summarize](https://urlpipe.dev/docs/summarize) call, 1 for an [/html](https://urlpipe.dev/docs/html) one, 0 for a cache hit or a failed analysis. `X-Quota-Limit` and `X-Quota-Remaining` answer "how much of my plan have I got left". A [/scrape](https://urlpipe.dev/docs/scrape) reports the **sum** of what its operations cost, in the same header — there is one allowance, so however many operations a request ran, it has one price. A request that asked for a [residential exit](https://urlpipe.dev/docs/residential) has that in the same number too, counted once per page fetch. The count is taken **after** your request, so `X-Quota-Remaining` is what you have left now — not what you had before the call. Cache hits are free, so a `hit` leaves it unchanged. On an unlimited plan both values read `unlimited` rather than a number. Running out of that allowance is not the end of the month on a paid plan — it is not the end of anything. `X-Quota-Overage` counts the credits spent past it: 0 until the allowance is gone, and after that the number that gets billed when the month closes. There is no companion header for how much further you can go, because on a plan with a card on file you are never cut off for credits. On the Free plan there is no overage and the allowance is the wall. See [overage](https://urlpipe.dev/docs/credits#overage). ## Sizing a worker pool `X-Concurrency-Limit` is how many requests your plan can run [in parallel](https://urlpipe.dev/docs/credits#concurrency). It is a property of the plan rather than a measurement, so it costs us nothing to send and it is the same on every response — which makes it the right number to size a worker pool from, instead of discovering the limit by being refused. There is deliberately no companion header counting what is in flight right now. It would be a database query on every response for a number that is already out of date by the time you read it; a client that tracks its own outstanding requests knows sooner and more accurately. The `concurrency_limit` 429 carries the live count when it actually matters. Back off before you run out ``` const res = await fetch("https://urlpipe.dev/markdown", { method: "POST", headers: { "Authorization": "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ url: "https://example.com", sync: true }), }) // What this call cost, and what is left afterwards. const cost = Number(res.headers.get("X-Quota-Cost")) const remaining = res.headers.get("X-Quota-Remaining") const overage = Number(res.headers.get("X-Quota-Overage")) // Past the included allowance: still being served — always — now at // $1.50 per 1,000 credits. Worth logging the first time it happens. if (overage > 0) { console.warn(`${overage} credits into overage this month`) } // Compare against the cost of the call you are about to make, not a // flat number: 17 credits is one more /summarize, or seventeen /html. if (remaining !== "unlimited" && Number(remaining) < cost) { // Raise max_age to lean on the cache, reach for a cheaper endpoint, // or wait for the reset. console.warn(`${remaining} credits left until ${res.headers.get("X-Quota-Reset")}`) } ``` A `429 quota_exceeded` carries the same quota headers, so you can read your allowance off one place whether the request was served or refused. The separate `429 rate_limited` response is a different thing — see [Credits & rate limits](https://urlpipe.dev/docs/credits). Source: https://urlpipe.dev/docs/response-headers --- # Errors URLpipe uses conventional HTTP status codes and a consistent JSON error shape, so failures are easy to detect and handle. ## Status codes | Status | Meaning | | --- | --- | | `200 OK` | The request succeeded. Async requests return 200 with a token. | | `401 Unauthorized` | Missing or invalid API key. | | `403 Forbidden` | The key is valid, but the email address on the account hasn't been confirmed. | | `422 Unprocessable Entity` | The operation failed or a parameter was invalid — see the error field. | | `429 Too Many Requests` | Too many of your requests already running (concurrency\_limit), sending too fast (rate\_limited), or — Free plan only — out of credits (quota\_exceeded). A paid plan is never refused for credits; it keeps serving at the overage rate. | ## Error shape When an operation fails, the response is JSON with a single `error` field holding a human-readable message: 422 Unprocessable Entity ``` { "error": "The request timed out." } ``` Three responses carry extra fields: the [quota](https://urlpipe.dev/docs/credits) 429 (with `limit`, `used`, `needed`, `resets_at`), the [parallel-requests](https://urlpipe.dev/docs/credits#concurrency) 429 (with `limit` and `running`), and the invalid-parameter 422 shown below. ## Authentication errors A `401 Unauthorized` means the `Authorization` header is missing or the token doesn't match an active project key. A `403 Forbidden` with the code `email_unverified` means the key is fine but the email address on the account has never been confirmed — resending the key or rotating it will not help; clicking the link we emailed you will. See [Authentication](https://urlpipe.dev/docs/authentication). ## Validation errors Bad parameters return a `422` with a machine-readable `error` code: - **Name** : `invalid_url` **Description** : The `url` is missing or not acceptable. It must be a valid `http`/`https` URL for a public domain — not an IP address, `localhost`, an internal hostname, or a custom (non-default) port. It must not carry credentials: `https://user:pass@example.com` is rejected. - **Name** : `invalid_max_age` **Description** : The `max_age` value couldn't be parsed. Use a number of seconds or a duration like `"2 hours"`. - **Name** : `invalid_options` **Description** : A [page option](https://urlpipe.dev/docs/page-options) or [screenshot option](https://urlpipe.dev/docs/screenshot#options) is out of range, of the wrong type or not one we know, and the message names which — `"screenshot_options.viewport_width must be a whole number from 320 to 1920."`, for example. - **Name** : `invalid_labels` **Description** : [labels](https://urlpipe.dev/docs/labels) must be an object of up to 16 keys with string values, and the message names what to change — `"labels.client must be a string."`, for example. - **Name** : `invalid_idempotency_key` **Description** : The [Idempotency-Key](https://urlpipe.dev/docs/retries#idempotency-key) must be 1 to 255 printable ASCII characters, with no spaces. - **Name** : `idempotency_key_reused` **Description** : This `Idempotency-Key` came with a different request in the last 24 hours. Nothing ran: send a new key for a new request. See [retries & duplicates](https://urlpipe.dev/docs/retries#errors). - **Name** : `report_to …` **Description** : A `report_to` was given that we will not deliver to, and the message names the reason — the same [rules the URL being analysed is held to](https://urlpipe.dev/docs/async#endpoint-rules), so an IP address, a `localhost` address, credentials in the URL or a custom port are all refused. Omitting it entirely is not an error: see [where results go](https://urlpipe.dev/docs/async#where-results-go). ## Common failure messages Operational failures — network issues, unreachable or oversized pages — come back as a `422` with one of these messages: | Message | Cause | | --- | --- | | The request timed out. | The page took too long to load. | | The connection to the server timed out. | A connection to the host could not be established in time. | | The requested page was not found. | The host couldn't be resolved, or the page returned HTTP 404. | | An internal server error occurred in the requested page. | The page returned an HTTP 5xx status. | | The request was invalid. | The page returned another 4xx status (e.g. 401, 403, 410). | | There was a problem with the SSL certificate. | The host's TLS certificate could not be validated. | | The server refused the connection. | The host actively refused the connection. | | The page is rate-limiting requests. | The page returned HTTP 429. We slow down and try again ourselves before you ever see this. | | Too many redirects occurred while processing the request. | A redirect loop was detected. | | No internet connection was detected. | A network connectivity problem occurred. | | The request was blocked by the client. | The request was blocked (e.g. by ad-blocking rules). | | The requested URL resolved to an address that is not publicly reachable. | The URL, or a redirect from it, pointed at a private or loopback address. Only public web addresses can be analysed. | | The requested URL is not a web page. | The URL returned something other than HTML — a PDF, an image or a download, for example. | | The page asked us to complete a bot check before it would load. | The site served an anti-bot challenge instead of the page. We work the challenge before you see this — waiting it out and retrying — and most of them clear; a handful of sites put a CAPTCHA in front of every visitor that is not a person. | | The page could not be loaded. | The page never came up, so there was nothing to analyse. | | An unexpected error occurred while processing the request. | Something failed that we do not have a specific answer for. These are reported to us automatically; retrying is usually worthwhile. | | The page is too big to be processed. | The HTML exceeds the 10 MB limit, or the page holds more content than an AI operation can return in one response. | | No residential exit was free to load this page. Retry shortly, or send the same request without residential to use our standard network. | The request asked for a residential exit and none could take it. There is no fallback here on purpose — answering from a datacentre address is the one thing the request ruled out. Nothing is spent; retry in a moment, or drop the parameter. | | A selector in the request is not valid CSS. | A selector in page\_options or screenshot\_options could not be parsed as CSS. | | No visible element on the page matched the selector. | A screenshot's selector matched nothing, or only an element with no size. Nothing is spent. | | The element named by wait\_for\_selector did not appear on the page. | The page was given 10 seconds for the element and it never arrived. Nothing is spent. | | The screenshot is too large to return. Lower full\_page\_max\_height, or use format jpeg or webp. | The image came out larger than 20 MB — usually a very long page at device\_scale\_factor 2 or 3, as a PNG. | | The site's robots.txt disallows this page, and this project is set to follow robots.txt. | The project follows robots.txt and a rule in the site's file covers this page, so it was not fetched. Nothing is spent. See robots.txt for how the rules are read and where the setting lives. | ## Busy pages are waited out for you Three of those messages — a timeout, a 5xx and a 429 — mean the page is struggling rather than broken, and they are the ones a retry actually fixes. URLpipe does that part itself: when a page starts stalling, requests to that host are spaced further and further apart and the analysis is tried again, for up to ten minutes, before any failure is reported. A crawl of a site that cannot keep up therefore takes longer and still finishes, instead of failing every request after the first few. What you see is a request that takes longer, not one that fails. A [synchronous](https://urlpipe.dev/docs/async) request may outlive its wait and answer `processing_timeout` with a token — collect the result from [GET /result/:token](https://urlpipe.dev/docs/results) as you would for any long analysis. An async request simply delivers later. The `processing_time_ms` we report covers the whole of it, waiting included. ## Recommended handling - Check the HTTP status first: `401` → fix credentials, `403` → confirm the account's email address, `429` → back off, `422` → inspect the message. - Retry timeouts and connection errors with backoff; they're often transient — and by the time one reaches you we have already [waited the page out](https://urlpipe.dev/docs/errors#busy-targets) without success. - Keep target pages under 10 MB of HTML to avoid `The page is too big to be processed.`. Source: https://urlpipe.dev/docs/errors --- # HTML Fetch the fully rendered HTML of any page. URLpipe loads the URL in headless Chrome, so JavaScript runs and client-rendered markup is included — not just the initial server response. This is the cheapest operation we sell: it uses no AI, and costs **1 credit** per call — the unit every other endpoint is priced against. See [Credits](https://urlpipe.dev/docs/credits). POST/html ## Get rendered HTML ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `page_options` **Type** : object **Description** : Wait for the page, and remove ads, cookie banners or your own elements from it before it is read — gone from this result, not merely hidden. See [Page options](https://urlpipe.dev/docs/page-options). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). ### Response Content type `text/plain` — the response body is the raw HTML document as a string. ``` curl -X POST https://urlpipe.dev/html \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` Response ``` Example Domain

Example Domain

This domain is for use in illustrative examples…

``` Need clean content instead of raw markup? [/markdown](https://urlpipe.dev/docs/markdown) strips navigation and boilerplate and returns the main content as Markdown. ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/rendered-html) Source: https://urlpipe.dev/docs/html --- # Screenshot Capture the whole rendered page as a full-page PNG — top to bottom, not just the fold — or size, crop and clean it up first: any viewport, retina, one element, JPEG or WebP, dark mode, and the ads and cookie banners out of the way. The image comes back Base64-encoded, ready to store or decode to a file, with a link to it in the response headers. It uses no AI, and costs **1 credit** per call — whichever options you use. See [Credits](https://urlpipe.dev/docs/credits). POST/screenshot ## Capture a screenshot ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `screenshot_options` **Type** : object **Description** : How to take the screenshot: size, format, what to wait for and what to hide. Every key is optional and included in the credit — see [Options](#options). Leave it out for a full-page PNG. - **Name** : `page_options` **Type** : object **Description** : Wait for the page, and remove ads, cookie banners or your own elements from it before it is read — gone from this result, not merely hidden. See [Page options](https://urlpipe.dev/docs/page-options). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). ### Response Content type `text/plain` — the body is the image, Base64-encoded: a PNG unless you asked for another `format`. Decode it to bytes, or prefix it with `data:image/png;base64,` (or `image/jpeg`, `image/webp`) to use it directly in an `` tag. Screenshots are **full-page** unless you ask otherwise: the whole scrollable document, not just what fits above the fold. The page is scrolled through first so lazy-loaded images are actually in the picture. The viewport is **1350 × 797** — a desktop browser window — unless you set one, so two captures of a page line up and can be compared. The height follows the page, up to a ceiling of **16,384 pixels**; a page taller than that is captured down to it. The response also carries an `X-Result-Url` header: a link to the same image that needs no API key, so it goes straight into an `` tag, an email or a social card. It stays valid for the 30 days the result is kept — see [Response headers](https://urlpipe.dev/docs/response-headers). ``` curl -X POST https://urlpipe.dev/screenshot \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "screenshot_options": {"format":"webp"}, "page_options": {"block_cookie_banners":true}}' ``` Response ``` UklGRlq4AABXRUJQVlA4WAoAAAAgAAAARQUAhAUA… (Base64-encoded WebP, truncated) ``` ## Options The keys of `screenshot_options`. Each one you leave out is its default, and a request that spells out a default is the same request as one that omits it — it is served the same [stored result](https://urlpipe.dev/docs/caching). A value out of range, or a key not listed here, is refused with `invalid_options` before anything is fetched or spent. Request body ``` { "url": "https://example.com/pricing", "screenshot_options": { "viewport_width": 390, "device_scale_factor": 2, "format": "webp", "hide_selectors": ["#chat-widget"] }, "page_options": { "block_cookie_banners": true } } ``` What the page is made to wait for or leave out — ads, cookie banners, an element that loads late — is set with [page\_options](https://urlpipe.dev/docs/page-options), which every endpoint that loads the page takes. ### Size and shape - **Name** : `full_page` **Type** : boolean **Description** : Capture the whole page top to bottom (the default), or set `false` for the viewport only. A viewport capture of a URL with a `#fragment` is scrolled to that anchor, the way a browser opens it. - **Name** : `full_page_max_height` **Type** : integer **Description** : The tallest the image may be, in CSS pixels, for a full page or an element: `100`–`16384` (the default). A page taller than this is captured down to it. - **Name** : `viewport_width` **Type** : integer **Description** : The browser window's viewport width, `320`–`1920`. Default `1350`. Use `390` to see a site's phone layout. - **Name** : `viewport_height` **Type** : integer **Description** : `240`–`1080`. Default `797`. Decides what the fold is, and the size of anything a page sizes to the screen. - **Name** : `device_scale_factor` **Type** : integer **Description** : Pixel density, `1`–`3`. `2` renders a retina image at twice the width and height, with the high-resolution images a retina display is sent. - **Name** : `selector` **Type** : string **Description** : A CSS selector: capture that one element instead of the page — a pricing table, a chart, a hero. The first match is used; a selector that matches nothing is a failed analysis and costs nothing. ### Format - **Name** : `format` **Type** : string **Description** : `png` (default), `jpeg` or `webp`. WebP is usually the smallest by far for a long page. - **Name** : `quality` **Type** : integer **Description** : JPEG and WebP only, `1`–`100`. Default `80`. A PNG is always lossless, so this has no effect on one. - **Name** : `omit_background` **Type** : boolean **Description** : Leave the background transparent where the page itself sets none. PNG and WebP only — a JPEG has no transparency. ### Look - **Name** : `dark_mode` **Type** : boolean **Description** : Render with `prefers-color-scheme: dark`, so a site that has a dark theme shows it. - **Name** : `hide_selectors` **Type** : array **Description** : CSS selectors of elements to hide in the image — a chat widget, a promo bar. Up to 50. They are hidden, not removed: to take them out of the page itself, use `page_options.remove_selectors`. - **Name** : `styles` **Type** : string **Description** : CSS to add to the page before capturing, up to 20,000 characters. ## Using the image URL If what you want is a link rather than bytes — a preview in a page you're rendering, an image in a digest email — read it off the response instead of decoding anything. The URL carries its own signed, expiring credentials, so it works anywhere, with no key attached and nothing to proxy. ``` curl -sD - -o /dev/null -X POST https://urlpipe.dev/screenshot \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "sync": true}' \ | grep -i '^x-result-url' ``` An async request has no response headers to read, so the same link arrives in the webhook payload as `result_url`, beside the `result` itself — see [Async mode](https://urlpipe.dev/docs/async). ## Decoding the image ``` curl -s -X POST https://urlpipe.dev/screenshot \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' \ | base64 --decode > screenshot.png ``` Screenshots can take a few seconds for heavy pages. For batches or slow sites, consider [async mode](https://urlpipe.dev/docs/async) so you're not holding a connection open. ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/website-screenshot) Source: https://urlpipe.dev/docs/screenshot --- # Markdown Convert a page's main content to clean, well-formatted Markdown. URLpipe keeps the substance — headings, paragraphs, lists, tables, links and fenced code — while dropping navigation, sidebars, cookie banners and other chrome. This is the go-to endpoint for feeding web pages to an LLM or a RAG pipeline: Markdown is the format models work best with, and because the page is rendered with headless Chrome first, JavaScript-heavy sites produce complete content too. The same page always produces the same Markdown, and links come back as absolute URLs you can follow without the page they came from. It uses no AI — the conversion is a walk over the rendered DOM, not a model call — so it costs **1 credit** per call, the same as [/html](https://urlpipe.dev/docs/html). See [Credits](https://urlpipe.dev/docs/credits). POST/markdown ## Convert to Markdown ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `page_options` **Type** : object **Description** : Wait for the page, and remove ads, cookie banners or your own elements from it before it is read — gone from this result, not merely hidden. See [Page options](https://urlpipe.dev/docs/page-options). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). ### Response Content type `text/plain` — the body is the page's main content as Markdown. ``` curl -X POST https://urlpipe.dev/markdown \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` Response ``` # Example Domain This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission. [More information...](https://www.iana.org/domains/example) ``` ## Size limit This endpoint accepts up to **10 MB** of HTML. Larger pages return a `422` with `The page is too big to be processed.` — see [Errors](https://urlpipe.dev/docs/errors). ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/url-to-markdown) Source: https://urlpipe.dev/docs/markdown --- # Metadata Extract structured metadata from a page. URLpipe reads the markup with AI and returns a tidy JSON object — title, description, language, author, publication date, feed URL and main image — normalising the messy variety of meta tags, JSON-LD and microdata found in the wild. This is an **AI** operation: it sends the page to a language model, so it costs **5 credits** per call. See [Credits](https://urlpipe.dev/docs/credits). POST/meta ## Extract metadata ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `page_options` **Type** : object **Description** : Wait for the page, and remove ads, cookie banners or your own elements from it before it is read — gone from this result, not merely hidden. See [Page options](https://urlpipe.dev/docs/page-options). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). ### Response fields URL fields are returned absolute, resolved against the requested page — a page that declares `/favicon.ico` comes back as a full URL. - **Name** : `title` **Type** : string **Description** : The page's main title. - **Name** : `description` **Type** : string **Description** : The page description. - **Name** : `language` **Type** : string **Description** : ISO 639-1 language code, e.g. `en`. - **Name** : `main_image_url` **Type** : string | null **Description** : Absolute URL of the main image; null if none or a data URI. - **Name** : `favicon_url` **Type** : string | null **Description** : Absolute URL of the favicon; null if none or a data URI. - **Name** : `author_name` **Type** : string | null **Description** : The content author, if any. - **Name** : `feed_url` **Type** : string | null **Description** : Absolute RSS or Atom feed URL, if discovered. - **Name** : `publication_date` **Type** : string | null **Description** : Date of FIRST publication in ISO 8601, if found. - **Name** : `additional_author_information` **Type** : object | null **Description** : Extra author details such as social handles or email, if found. ``` curl -X POST https://urlpipe.dev/meta \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` Response ``` { "title": "Example Domain", "description": "Illustrative examples in documents.", "language": "en", "main_image_url": "https://example.com/cover.jpg", "favicon_url": "https://example.com/favicon.ico", "author_name": "Jane Doe", "feed_url": "https://example.com/feed.xml", "publication_date": "2026-01-01T00:00:00Z", "additional_author_information": { "twitter": "@janedoe", "email": "jane@example.com" } } ``` Any field can be `null` when the information isn't present on the page. AI endpoints accept up to 10 MB of HTML. ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/metadata-extractor) Source: https://urlpipe.dev/docs/meta --- # Summarize Get a concise, AI-generated summary of a page's main content, formatted as Markdown. Useful for previews, digests, newsletters and triaging long articles. This is an **AI** operation: it sends the page to a language model, so it costs **17 credits** per call. See [Credits](https://urlpipe.dev/docs/credits). POST/summarize ## Summarize a page ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `page_options` **Type** : object **Description** : Wait for the page, and remove ads, cookie banners or your own elements from it before it is read — gone from this result, not merely hidden. See [Page options](https://urlpipe.dev/docs/page-options). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). ### Response Content type `text/plain` — a Markdown summary, typically three to five short paragraphs. ``` curl -X POST https://urlpipe.dev/summarize \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` Response ``` The page introduces the Example Domain, a reserved domain name set aside for use in documentation and illustrative examples. It explains that the domain can be referenced in articles and other literature without needing prior permission or coordination, and links to further information maintained by IANA. ``` AI endpoints accept up to 10 MB of HTML. For the full content rather than a summary, use [/markdown](https://urlpipe.dev/docs/markdown). ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/summarize-url) Source: https://urlpipe.dev/docs/summarize --- # Keywords Extract the keywords and key phrases that best describe a page. URLpipe returns 5–15 terms as a JSON array, ordered by relevance — handy for tagging, categorisation, related-content and search. This is an **AI** operation: it sends the page to a language model, so it costs **15 credits** per call. See [Credits](https://urlpipe.dev/docs/credits). POST/keywords ## Extract keywords ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `page_options` **Type** : object **Description** : Wait for the page, and remove ads, cookie banners or your own elements from it before it is read — gone from this result, not merely hidden. See [Page options](https://urlpipe.dev/docs/page-options). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). ### Response Content type `application/json` — an array of strings, most relevant first. ``` curl -X POST https://urlpipe.dev/keywords \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` Response ``` [ "example domain", "documentation", "illustrative examples", "reserved domain", "IANA" ] ``` AI endpoints accept up to 10 MB of HTML. ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/keyword-extractor) Source: https://urlpipe.dev/docs/keywords --- # Lighthouse Run a real Google Lighthouse audit against a page and get back the category scores and key performance metrics as JSON — with an option to include the full set of 150+ audits for deep diagnostics. It uses no AI, and costs **2 credits** per call — a full audit holds a browser for far longer than a page fetch does. See [Credits](https://urlpipe.dev/docs/credits). Audits can take a while, so this one pairs well with [async mode](https://urlpipe.dev/docs/async). POST/lighthouse ## Run an audit ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `device` **Type** : string **Description** : Device to emulate: `mobile` (default) or `desktop`. - **Name** : `include_audits` **Type** : string **Description** : Set to `"true"` to include the full `audits` object (150+ audits). Defaults to `"false"`. - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). ``` curl -X POST https://urlpipe.dev/lighthouse \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "device": "desktop"}' ``` Response ``` { "url": "https://example.com", "fetchTime": "2026-01-01T00:00:00.000Z", "device": "desktop", "categories": { "performance": { "score": 0.95, "title": "Performance" }, "accessibility": { "score": 0.88, "title": "Accessibility" }, "best-practices":{ "score": 0.92, "title": "Best Practices" }, "seo": { "score": 0.90, "title": "SEO" }, "pwa": null }, "metrics": { "first-contentful-paint": { "score": 0.95, "displayValue": "1.2 s", "numericValue": 1200, "numericUnit": "millisecond" }, "largest-contentful-paint": { "score": 0.90, "displayValue": "2.5 s", "numericValue": 2500, "numericUnit": "millisecond" }, "cumulative-layout-shift": { "score": 1.0, "displayValue": "0.05", "numericValue": 0.05, "numericUnit": "unitless" }, "total-blocking-time": { "score": 0.93, "displayValue": "150 ms", "numericValue": 150, "numericUnit": "millisecond" } } } ``` ## Category scores Each category has a `score` from `0` to `1`. The `pwa` key is always `null`: Lighthouse no longer audits Progressive Web Apps, and the key stays so the response keeps one shape. Categories reported: `performance`, `accessibility`, `best-practices` and `seo`. ## Metrics Each metric includes `score`, `displayValue`, `numericValue` and `numericUnit`. Any metric may be `null` if Lighthouse couldn't compute it. ## Device emulation - **Name** : `mobile` **Type** : default **Description** : 360×640 viewport, 4× CPU slowdown, slow-4G network. Simulates real-world mobile conditions and typically produces lower scores. - **Name** : `desktop` **Description** : 1350×940 viewport, no CPU throttling, fast network. Generally produces higher scores. ## Core Web Vitals & INP LCP and CLS are included in `metrics`. INP (Interaction to Next Paint) is a **field** metric that needs real user interactions and can't be measured in a lab test like Lighthouse. Use `total-blocking-time` (TBT) as the lab proxy for responsiveness — it correlates strongly with INP. ## Full audit data Pass `include_audits: "true"` to add an `audits` object with all 150+ Lighthouse audits — performance diagnostics (`render-blocking-resources`, `unused-css-rules`), accessibility checks (`color-contrast`, `image-alt`), SEO audits and best-practices. Each audit carries a score, title, description and often a `details` object listing the specific elements to fix. The mobile and desktop audits — and audits with and without `include_audits` — are cached separately, since those options change the result. See [Caching](https://urlpipe.dev/docs/caching). ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/lighthouse-audit) Source: https://urlpipe.dev/docs/lighthouse --- # Console Capture the JavaScript console output a page produces as it loads — errors, warnings and uncaught exceptions. Useful for monitoring third-party scripts, catching regressions and health-checking your own pages. It uses no AI, and costs **1 credit** per call. See [Credits](https://urlpipe.dev/docs/credits). POST/console ## Capture console messages ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `page_options` **Type** : object **Description** : Wait for the page, and remove ads, cookie banners or your own elements from it before it is read — gone from this result, not merely hidden. See [Page options](https://urlpipe.dev/docs/page-options). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). ### Response Content type `application/json` — an array of message objects. An empty array `[]` means the page produced no errors or warnings. ### Message fields - **Name** : `type` **Type** : string **Description** : One of `error`, `warning` or `exception`. - **Name** : `text` **Type** : string **Description** : The message text. ``` curl -X POST https://urlpipe.dev/console \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}' ``` Response ``` [ { "type": "error", "text": "Cart failed to load: TypeError: Failed to fetch" }, { "type": "warning", "text": "[analytics] consent not given, events are queued" }, { "type": "exception", "text": "ReferenceError: bar is not defined" } ] ``` ## Message types - **Name** : `error` **Description** : From `console.error()` calls. - **Name** : `warning` **Description** : From `console.warn()` calls. - **Name** : `exception` **Description** : Uncaught JavaScript exceptions, and promise rejections nothing handled. Plain `console.log()` output is not captured. ## How it works URLpipe loads the page in a headless browser, listens for console errors, warnings and uncaught exceptions, then waits briefly after the page's network activity settles so asynchronous scripts have time to run and report. All captured messages are returned together. Messages come from the page itself. Scripts inside embedded frames — ads, widgets, third-party players — log to their own console, so what you get back is your page's output rather than everyone else's. ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/console-errors) Source: https://urlpipe.dev/docs/console --- # Scrape The fastest way to get several extractions from the same URL: one request, one page visit. Every operation shares a single page load — the slowest part of any analysis — instead of each paying for its own, and the AI extractions run in parallel on top of it. **Speed is the whole point of this endpoint.** The results themselves are identical to the individual endpoints', and billing is unchanged too: the request spends the sum of what its operations [cost](https://urlpipe.dev/docs/credits), and operations served from cache stay free. You pay the same — you just get everything much sooner. There is one exception, and it is in your favour. A [residential exit](https://urlpipe.dev/docs/residential) is charged per page fetch, and a scrape fetches once, so `residential: true` costs one exit here where five separate calls would cost five. (`lighthouse` runs its own audit on its own engine, so a scrape including it fetches twice, and so does a screenshot with an option that changes how the page loads — see [How it works](#how).) POST/scrape ## Run multiple operations at once ### Body parameters - **Name** : `url` **Type** : string **Required** : Required **Description** : The absolute URL of the page to process. Rendered with headless Chrome, so JavaScript runs and redirects are followed. It must not include a username or password (`https://user:pass@example.com`). - **Name** : `operations` **Type** : array **Required** : Required **Description** : The operations to run — any non-empty subset of `html`, `markdown`, `meta`, `summarize`, `keywords`, `screenshot`, `console` and `lighthouse`. A JSON array (`["markdown","meta"]`) or a comma-separated string (`markdown,meta`). An unknown operation returns `422`. - **Name** : `page_options` **Type** : object **Description** : Wait for the page, and remove ads, cookie banners or your own elements from it before it is read — gone from this result, not merely hidden. See [Page options](https://urlpipe.dev/docs/page-options). - **Name** : `residential` **Type** : boolean **Description** : Fetch the page from a [residential exit](https://urlpipe.dev/docs/residential) — an address on a home broadband line rather than one in a datacentre. Reach for it when a site serves you less than it serves a browser, or nothing at all. Defaults to `false`. Adds **25 credits** per page fetch on top of what the operation costs, and its results are kept separate from the ordinary ones. - **Name** : `report_to` **Type** : string **Description** : Webhook URL — an `http` or `https` address URLpipe POSTs the result to when it's ready. **Optional**: without it we deliver to your project's [default endpoint](https://urlpipe.dev/docs/async#where-results-go) if it has one, and otherwise send no webhook at all — the result still waits for you at [GET /result/:token](https://urlpipe.dev/docs/results). A value we cannot deliver to returns `422`. Ignored on a `sync=true` request. Deliveries can be [signed](https://urlpipe.dev/docs/async#signature) so your endpoint can verify they came from us. - **Name** : `sync` **Type** : boolean **Description** : Process the request synchronously, returning the result inline in the response. Defaults to `false` (async: return a token now, and either receive the result at a [webhook](https://urlpipe.dev/docs/async#where-results-go) or fetch it with [GET /result/:token](https://urlpipe.dev/docs/results)). See [Async & sync modes](https://urlpipe.dev/docs/async) for the full contract. - **Name** : `max_age` **Type** : string | integer **Description** : How fresh a cached result must be to be accepted. Either an **integer** number of seconds (`3600`) or a **duration string** of the form `" "` — units `s`/`min`/`h`/`d`/`w` (e.g. `"2 hours"`, `"3 days"`, `"30m"`). Defaults to `7 days`, clamped to a max of `30 days`; `0` always bypasses the cache. See [Caching](https://urlpipe.dev/docs/caching) for all accepted units. - **Name** : `labels` **Type** : object **Description** : Your own keys to find and account for this request by — a client, a project, a campaign: `{"client": "acme"}`. Returned with the result, in the webhook and in the `X-Labels` header, and your dashboard filters history and totals credits by them. Up to 16 keys; string values. See [Labels](https://urlpipe.dev/docs/labels). - **Name** : `include_audits` **Type** : boolean **Description** : Only used when `operations` includes `lighthouse`: include the detailed audits section in its result. Defaults to `false`. - **Name** : `device` **Type** : string **Description** : Only used when `operations` includes `lighthouse`: the device profile to audit with, `mobile` (default) or `desktop`. - **Name** : `screenshot_options` **Type** : object **Description** : Only used when `operations` includes `screenshot`: how to take it, with the same [keys](https://urlpipe.dev/docs/screenshot#options) `/screenshot` takes. ### Response Content type `application/json` — one entry per requested operation, each with its own `success` flag. A failing operation (for example a page too large for an AI extraction) never affects its siblings; each entry carries either its `result` in that operation's usual format or its `error`. `cached` tells you when an operation was served from cache (free). ``` curl -X POST https://urlpipe.dev/scrape \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "operations": ["markdown","meta","screenshot"], "sync": "true"}' ``` Response ``` { "url": "https://example.com", "operations": { "markdown": { "success": true, "result": "# Example Domain\n\nThis domain is for use in…", "cached": false }, "meta": { "success": true, "result": { "title": "Example Domain", "description": "…", "language": "en" }, "cached": true }, "screenshot": { "success": true, "result": "iVBORw0KGgoAAAANSUhEUg…", "cached": false } } } ``` ## How it works URLpipe loads the page once in a headless browser and captures, from that single visit, everything the requested operations need: the rendered HTML (which feeds `html`, `markdown`, `meta`, `summarize` and `keywords`), the `screenshot` and the `console` messages. The AI extractions then run in parallel. Every result is identical to what the individual endpoint would return, and is cached under the same key — a scrape can be served by earlier individual calls, and later individual calls can be served by a scrape. The one exception is `lighthouse`: a performance audit needs its own instrumented page load, so it runs alongside the unified visit and its result is merged into the combined response when it finishes. A screenshot shares the visit too, with most of its `screenshot_options`: a format, a selector and hidden elements all apply to the image alone. Four change how the page itself loads — `viewport_width`, `viewport_height`, `device_scale_factor` and `dark_mode` — so a screenshot with any of them loads the page again on its own, and the `html` and `markdown` beside it stay the page as it normally renders. [page\_options](https://urlpipe.dev/docs/page-options) apply to the shared visit, and so to every operation of the scrape except `lighthouse`, whose audit loads the page for itself. Including `lighthouse`? Prefer **async mode** — audits regularly outlast the 60-second synchronous window, in which case the sync response is a `504` with a token to fetch later. Also note that a combined response embeds the screenshot as Base64, so webhook payloads can be several megabytes. ## Partial failures Operations fail independently: one failed extraction leaves the others intact, and the response is still `200 OK`. Only when every operation fails — typically because the page itself was unreachable — is the whole request a `422`. Failed operations never spend credits. If a lighthouse audit is still running when the rest of the scrape finishes, its entry reports `processing_timeout`. It keeps running — fetch the scrape again from [GET /result/:token](https://urlpipe.dev/docs/results) with this request's token and the entry will be filled in once it lands. The individual operations have no tokens of their own; the scrape is addressed as a whole. ## Responses Whatever the status, the response carries [metadata headers](https://urlpipe.dev/docs/response-headers): the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left. Status When Body `200 OK` The request succeeded. Sync: the result, in this endpoint's format (see Response above). Async: a job token and the request's labels — { "token": "…", "status": "accepted", "labels": {} }. `422 Unprocessable Entity` The analysis failed, or a parameter was invalid (a bad max\_age, a screenshot\_options value out of range, labels that break the rules, or a report\_to we will not deliver to). { "error": "" } `429 Too Many Requests` Three causes, told apart by the error field: concurrency\_limit (too many of your requests already running), rate\_limited (sending too fast), or quota\_exceeded (Free plan only — out of credits with no card on file to bill the extra to, and checked only on a cache miss; a paid plan keeps serving at the overage rate and is never refused for credits). All three carry "error" and "message". Extra fields: concurrency\_limit → "limit", "running" · rate\_limited → "retry\_after" · quota\_exceeded → "limit", "used", "needed", "resets\_at". `504 Gateway Timeout` Sync only: the analysis didn't finish within 60s. It keeps running — fetch it via GET /result/:token. { "error": "processing\_timeout", "token": "…" } `401 Unauthorized` Missing or invalid API key. — Try it live — no API key needed Run this endpoint against any URL right in your browser. [Open tool](https://urlpipe.dev/tools/analyze-url) Source: https://urlpipe.dev/docs/scrape