Scrape

The fastest way to get several extractions from the same URL: one request, one page visit. Every operation shares a single page load — the slowest part of any analysis — instead of each paying for its own, and the AI extractions run in parallel on top of it.

Speed is the whole point of this endpoint. The results themselves are identical to the individual endpoints', and billing is unchanged too: each requested operation counts against its own quota category (AI or web), and operations served from cache stay free. You pay the same — you just get everything much sooner.

curl -X POST https://urlpipe.dev/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "operations": ["markdown","meta","screenshot"], "sync": "true"}'

const res = await fetch("https://urlpipe.dev/scrape", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com", operations: ["markdown","meta","screenshot"], sync: "true" }),
})

import requests

res = requests.post(
    "https://urlpipe.dev/scrape",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"url": "https://example.com", "operations": ["markdown","meta","screenshot"], "sync": "true"},
)

{
  "url": "https://example.com",
  "operations": {
    "markdown": {
      "success": true,
      "result": "# Example Domain\n\nThis domain is for use in…",
      "cached": false
    },
    "meta": {
      "success": true,
      "result": { "title": "Example Domain", "description": "…", "language": "en" },
      "cached": true
    },
    "screenshot": {
      "success": true,
      "result": "iVBORw0KGgoAAAANSUhEUg…",
      "cached": false
    }
  }
}

How it works

URLpipe loads the page once in a headless browser and captures, from that single visit, everything the requested operations need: the rendered HTML (which feeds html, markdown, meta, summarize and keywords), the screenshot and the console messages. The AI extractions then run in parallel. Every result is identical to what the individual endpoint would return, and is cached under the same key — a scrape can be served by earlier individual calls, and later individual calls can be served by a scrape.

The one exception is lighthouse: a performance audit needs its own instrumented page load, so it runs alongside the unified visit and its result is merged into the combined response when it finishes.

Partial failures

Operations fail independently: one failed extraction leaves the others intact, and the response is still 200 OK. Only when every operation fails — typically because the page itself was unreachable — is the whole request a 422. Failed operations never count toward your quota. If a lighthouse audit is still running when the rest of the scrape finishes, its entry reports processing_timeout. It keeps running — fetch the scrape again from GET /result/:token with this request's token and the entry will be filled in once it lands. The individual operations have no tokens of their own; the scrape is addressed as a whole.

Responses