Scrape

The fastest way to get several extractions from the same URL: one request, one page visit. Every operation shares a single page load — the slowest part of any analysis — instead of each paying for its own, and the AI extractions run in parallel on top of it.

Speed is the whole point of this endpoint. The results themselves are identical to the individual endpoints', and billing is unchanged too: the request spends the sum of what its operations cost, and operations served from cache stay free. You pay the same — you just get everything much sooner.

There is one exception, and it is in your favour. A residential exit is charged per page fetch, and a scrape fetches once, so residential: true costs one exit here where five separate calls would cost five. (lighthouse runs its own audit on its own engine, so a scrape including it fetches twice, and so does a screenshot with an option that changes how the page loads — see How it works.)

curl -X POST https://urlpipe.dev/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "operations": ["markdown","meta","screenshot"], "sync": "true"}'

import requests

res = requests.post(
    "https://urlpipe.dev/scrape",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"url": "https://example.com", "operations": ["markdown", "meta", "screenshot"], "sync": "true"},
)

const res = await fetch("https://urlpipe.dev/scrape", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com", operations: ["markdown","meta","screenshot"], sync: "true" }),
})

$ch = curl_init("https://urlpipe.dev/scrape");
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => [
        "Authorization: Bearer YOUR_API_KEY",
        "Content-Type: application/json",
    ],
    CURLOPT_POSTFIELDS => json_encode(["url" => "https://example.com", "operations" => ["markdown", "meta", "screenshot"], "sync" => "true"]),
]);

$response = curl_exec($ch);

{
  "url": "https://example.com",
  "final_url": "https://example.com/",
  "operations": {
    "markdown": {
      "success": true,
      "result": "# Example Domain\n\nThis domain is for use in…",
      "cached": false
    },
    "meta": {
      "success": true,
      "result": { "title": "Example Domain", "description": "…", "language": "en" },
      "cached": true
    },
    "screenshot": {
      "success": true,
      "result": "iVBORw0KGgoAAAANSUhEUg…",
      "cached": false
    }
  }
}

How it works

URLpipe loads the page once in a headless browser and captures, from that single visit, everything the requested operations need: the rendered HTML (which feeds html, markdown, meta, summarize and keywords), the screenshot and the console messages. The AI extractions then run in parallel. Every result is identical to what the individual endpoint would return, and is cached under the same key — a scrape can be served by earlier individual calls, and later individual calls can be served by a scrape.

The one exception is lighthouse: a performance audit needs its own instrumented page load, so it runs alongside the unified visit and its result is merged into the combined response when it finishes.

A screenshot shares the visit too, with most of its screenshot_options: a format, a selector and hidden elements all apply to the image alone. Four change how the page itself loads — viewport_width, viewport_height, device_scale_factor and dark_mode — so a screenshot with any of them loads the page again on its own, and the html and markdown beside it stay the page as it normally renders.

page_options apply to the shared visit, and so to every operation of the scrape except lighthouse, whose audit loads the page for itself.

Partial failures

Operations fail independently: one failed extraction leaves the others intact, and the response is still 200 OK. Only when every operation fails — typically because the page itself was unreachable — is the whole request a 422. Failed operations never spend credits. If a lighthouse audit is still running when the rest of the scrape finishes, its entry reports processing_timeout. It keeps running — fetch the scrape again from GET /result/:token with this request's token and the entry will be filled in once it lands. The individual operations have no tokens of their own; the scrape is addressed as a whole.

Responses

Whatever the status, the response carries metadata headers: the result token, whether it was served from cache and how old that result is, how long we took, what it cost in credits, and the allowance you have left.