# URLpipe

> URLpipe is a per-URL web data API: POST a URL and get back clean Markdown, rendered HTML, a full-page screenshot, page metadata, an AI summary, keywords, a Lighthouse audit or the JavaScript console output — or several of them from one page visit. Pages are rendered in real Chrome by our own rendering engine, so JavaScript-heavy sites work. One bearer token, no SDK, and an MCP server for AI agents.

URLpipe is for developers who would otherwise write and maintain one-off scrapers: feeding web pages to LLMs and RAG pipelines, giving AI agents a browser, link previews, and performance or error checks on sites you don't control. It does not crawl whole sites or discover URLs — it answers questions about the URL you give it.

Authentication: `Authorization: Bearer YOUR_API_KEY` (a per-project key) on every request. Each request is a POST with a JSON body containing `url`. Requests are asynchronous by default — you get a token back and collect the result at your webhook or with GET /result/:token — and `sync: true` returns the result inline.

## Endpoints

- [POST /html](https://urlpipe.dev/docs/html): Rendered HTML of a page (JavaScript executed, redirects followed). 1 credit.
- [POST /markdown](https://urlpipe.dev/docs/markdown): The page's main content as clean Markdown, converted deterministically from the rendered DOM — no model. 1 credit.
- [POST /screenshot](https://urlpipe.dev/docs/screenshot): Full-page screenshot as PNG, JPEG or WebP; viewport, retina scale, one element, dark mode, hidden elements and custom CSS via `screenshot_options`. 1 credit.
- [POST /meta](https://urlpipe.dev/docs/meta): Title, description, language, main image, favicon, author, publication date and feed, reconciled from Open Graph, Twitter cards and standard tags. 5 credits.
- [POST /summarize](https://urlpipe.dev/docs/summarize): A concise AI summary of the page's main content, in Markdown. 17 credits.
- [POST /keywords](https://urlpipe.dev/docs/keywords): 5–15 keywords and phrases, ranked by relevance, in the page's language. 15 credits.
- [POST /lighthouse](https://urlpipe.dev/docs/lighthouse): A Google Lighthouse audit (mobile or desktop): four category scores and the lab metrics. 2 credits.
- [POST /console](https://urlpipe.dev/docs/console): console.error and console.warn output, uncaught exceptions and unhandled promise rejections during page load. 1 credit.
- [POST /scrape](https://urlpipe.dev/docs/scrape): Any subset of the operations above from a single page visit. Costs the sum of the operations it runs.
- [GET /result/:token](https://urlpipe.dev/docs/results): Collect a result by its token, for 30 days.

## MCP server

URLpipe is also a remote MCP server at https://urlpipe.dev/mcp (streamable HTTP, `Authorization: Bearer YOUR_TOKEN` with an organization token — nothing to install). Its tools are the endpoints above, with the same arguments, responses and credits, plus tools to list projects, read usage and retrieve past results. Tokens can be read-only, limited to one project, or set to expire. Setup: https://urlpipe.dev/docs/mcp

## Features

- Caching: repeated requests for the same URL and operation within `max_age` (default 7 days, up to 30) are served from cache and spend no credits; a repeat that arrives while the first is still running waits for it, also free. https://urlpipe.dev/docs/caching
- Never pay twice: `Idempotency-Key` makes a retried request return the original answer. https://urlpipe.dev/docs/retries
- Async by default: webhooks retried with backoff and re-sendable from the dashboard, HMAC-SHA256 signed (`X-URLpipe-Signature`) once signing is switched on for the project; or poll GET /result/:token. https://urlpipe.dev/docs/async
- Page options: wait for an element or a delay, block ads, remove cookie banners and any elements you name — removed from every result, not just hidden. https://urlpipe.dev/docs/page-options
- Labels: tag requests with up to 16 of your own ids (client, project, job); they come back with results and webhooks, and usage is broken down by label. https://urlpipe.dev/docs/labels
- Residential exits: `residential: true` fetches from a home broadband address, +25 credits per page visit, off by default. https://urlpipe.dev/docs/residential
- robots.txt: every project follows the robots.txt of the sites it fetches by default; a customer can turn that off per project, and is then responsible for having the right to fetch those pages. https://urlpipe.dev/docs/robots-txt
- EU: pages are fetched, rendered and stored in the EU on every plan, and AI processing can be switched to EU-only per organization at no charge. https://urlpipe.dev/docs/data-residency
- Failed requests spend nothing: target errors, bot checks that couldn't be cleared, robots.txt refusals. https://urlpipe.dev/docs/credits

## Pricing

Sold in credits. Per call: /console 1, /html 1, /markdown 1, /screenshot 1, /lighthouse 2, /meta 5, /keywords 15, /summarize 17; /scrape is the sum of its operations; `residential: true` adds 25 per page visit. Cache hits and failures are free.

- Free: 1,000 credits/month, 1 in parallel, $0, no card.
- Starter: 20,000 credits/month, 3 in parallel, $19/month or $190/year.
- Pro: 55,000 credits/month, 8 in parallel, $49/month or $490/year.
- Scale: 175,000 credits/month, 20 in parallel, $149/month or $1,490/year.

Paid plans are never cut off: past the allowance, extra credits cost $1.50 per 1,000 credits, invoiced after the month closes. The Free plan is refused (HTTP 429) until the 1st. Prices are in USD before VAT. Full detail: https://urlpipe.dev/pricing.md

## Documentation

- [Introduction](https://urlpipe.dev/docs): URLpipe API docs: turn any URL into clean, structured data
- [Quickstart](https://urlpipe.dev/docs/quickstart): URLpipe Quickstart: your first API request in minutes
- [Authentication](https://urlpipe.dev/docs/authentication): Authentication: API keys & bearer tokens — URLpipe docs
- [MCP server](https://urlpipe.dev/docs/mcp): MCP server: connect an AI agent to the URLpipe API
- [Caching & freshness](https://urlpipe.dev/docs/caching): Caching & freshness: max_age and free cache hits — URLpipe
- [Async & sync modes](https://urlpipe.dev/docs/async): Async & sync modes: webhooks or inline — URLpipe API docs
- [Retries & duplicates](https://urlpipe.dev/docs/retries): Retries & idempotency keys: never pay twice — URLpipe
- [Retrieving results](https://urlpipe.dev/docs/results): Retrieving async results by token — URLpipe API docs
- [Page options](https://urlpipe.dev/docs/page-options): Page options: block ads, cookie banners, wait — URLpipe
- [Labels](https://urlpipe.dev/docs/labels): Labels: tag requests with your own ids — URLpipe docs
- [Residential exits](https://urlpipe.dev/docs/residential): Residential proxy exits: fetch from a home IP — URLpipe
- [robots.txt](https://urlpipe.dev/docs/robots-txt): robots.txt: follow or ignore crawl rules — URLpipe docs
- [Credits & rate limits](https://urlpipe.dev/docs/credits): Credits & rate limits: what each call costs — URLpipe docs
- [Data residency](https://urlpipe.dev/docs/data-residency): EU data residency: keep requests in Europe — URLpipe
- [Response headers](https://urlpipe.dev/docs/response-headers): Response headers: cache status, timing & quota — URLpipe
- [Errors](https://urlpipe.dev/docs/errors): API error responses & HTTP status codes — URLpipe docs
- [HTML](https://urlpipe.dev/docs/html): HTML endpoint: fetch a page's rendered HTML — URLpipe API
- [Screenshot](https://urlpipe.dev/docs/screenshot): Screenshot endpoint: full-page PNG, JPEG, WebP — URLpipe
- [Markdown](https://urlpipe.dev/docs/markdown): Markdown endpoint: convert a web page to Markdown — URLpipe
- [Metadata](https://urlpipe.dev/docs/meta): Metadata endpoint: title, description, author — URLpipe API
- [Summarize](https://urlpipe.dev/docs/summarize): Summarize endpoint: AI summary of any page — URLpipe API
- [Keywords](https://urlpipe.dev/docs/keywords): Keywords endpoint: extract page keywords — URLpipe API
- [Lighthouse](https://urlpipe.dev/docs/lighthouse): Lighthouse endpoint: Google performance audit — URLpipe
- [Console](https://urlpipe.dev/docs/console): Console endpoint: capture JS console logs — URLpipe API
- [Scrape](https://urlpipe.dev/docs/scrape): Scrape endpoint: multiple analyses in one call — URLpipe

## Free tools (live demos, no signup)

- [Turn any URL into clean Markdown](https://urlpipe.dev/tools/url-to-markdown): Paste a link and get the page's main content as tidy Markdown — headings, lists, links and code kept, navigation and cookie banners stripped. The format LLMs and RAG pipelines work best with.
- [Screenshot any website from its URL](https://urlpipe.dev/tools/website-screenshot): Paste a link and get a full-page PNG of the rendered page — JavaScript executed, exactly as a real browser would draw it. Great for previews, monitoring and visual QA.
- [Run a Lighthouse audit on any URL](https://urlpipe.dev/tools/lighthouse-audit): Paste a link and get a real Google Lighthouse audit — performance, accessibility, best practices and SEO scores, plus Core Web Vitals — measured against the live page.
- [Check any website for JavaScript errors](https://urlpipe.dev/tools/console-errors): Paste a link and see the console errors, warnings and uncaught exceptions logged while the page loads in a real browser — no DevTools, no local setup.
- [Check a page's Open Graph tags and metadata](https://urlpipe.dev/tools/metadata-extractor): Paste a link and get the page's metadata — title, description, author, publication date, feed and main image — cleaned into one structured object.
- [Summarize any web page with AI](https://urlpipe.dev/tools/summarize-url): Paste a link and get a concise AI summary of the page's main content — the substance, without the navigation, ads and boilerplate.
- [Get the keywords from any URL](https://urlpipe.dev/tools/keyword-extractor): Paste a link to any website or article and get the 5–15 keywords and phrases that describe it best, ranked — for SEO briefs, tagging and content analysis.
- [Get the rendered HTML of any URL](https://urlpipe.dev/tools/rendered-html): Paste a link and get the page's HTML after JavaScript has run and redirects have been followed — the DOM a real browser sees, not the empty shell curl returns.
- [Everything about a URL, from one visit](https://urlpipe.dev/tools/analyze-url): Paste a link and get a screenshot of the page, its title, description and share image, the main content as Markdown and the console errors it threw — all from the same page load.
- [See how a link unfurls in Slack, X, LinkedIn and Discord](https://urlpipe.dev/tools/link-preview-generator): Paste a link and see the card each app builds from its title, description and share image — with a screenshot of the page standing in when it has no image of its own.
- [Check a page's Core Web Vitals](https://urlpipe.dev/tools/core-web-vitals-checker): Paste a link and get Largest Contentful Paint, Cumulative Layout Shift and Total Blocking Time from a real Lighthouse run — each marked good, needs improvement or poor.
- [Take a full-page screenshot of any website](https://urlpipe.dev/tools/website-screenshot/full-page): Paste a link and get the entire page in one PNG — every section down to the footer, not just what fits on the screen — with lazy-loaded images loaded before the capture.
- [Screenshot a website the way a phone shows it](https://urlpipe.dev/tools/website-screenshot/mobile): Paste a link and get a full-page capture at a 390 px wide viewport — a common phone width — so you see the site's mobile layout, not its desktop one.
- [Screenshot a website in dark mode](https://urlpipe.dev/tools/website-screenshot/dark-mode): Paste a link and get a full-page capture rendered with the browser set to dark — so a site that ships a dark theme shows it, exactly as a visitor with dark mode on sees it.
- [Screenshot one element of a page by CSS selector](https://urlpipe.dev/tools/website-screenshot/element): Paste a link and a CSS selector and get just that element — a header, a pricing table, a chart — cropped to its own box, rendered in real Chrome.
- [Check whether robots.txt lets a bot fetch a URL](https://urlpipe.dev/tools/robots-txt-checker): Paste a link and see what the site's robots.txt says about that exact path — for URLpipe's user agent and for every crawler that has no group of its own — and which rule decides it.
- [Compare two websites side by side](https://urlpipe.dev/tools/compare-urls): Paste two links and get a screenshot of each with its Lighthouse scores and Core Web Vitals next to the other's — your page against a competitor's, or before against after.

## Guides

- [HTML to Markdown for LLMs](https://urlpipe.dev/guides/html-to-markdown-for-llms): Why Markdown is the right format for language models, what a clean conversion keeps and drops, and how to handle client-rendered pages.
- [Reading a Lighthouse audit](https://urlpipe.dev/guides/how-to-read-a-lighthouse-audit): What the four categories and the Performance score mean, how Core Web Vitals are measured, and how to prioritize fixes.
- [Rendered vs. raw HTML](https://urlpipe.dev/guides/rendered-vs-raw-html): Why modern sites return an empty shell to curl, how client-side rendering works, and how to get the post-JavaScript DOM.
- [Extracting page metadata](https://urlpipe.dev/guides/extract-open-graph-metadata): The three metadata standards, why they conflict in practice, and how to reconcile them into one clean object.
- [Why AI agents read empty pages](https://urlpipe.dev/guides/why-ai-agents-read-empty-pages): The reference fetch server does a plain GET, so client-rendered pages reach your agent empty — measured on three real pages.
- [Web access for Claude and Cursor](https://urlpipe.dev/guides/give-claude-and-cursor-web-access-with-mcp): Exact configuration for Claude Code, Claude Desktop and Cursor, token scopes, first prompts and costs.
- [Get keywords from a URL](https://urlpipe.dev/guides/get-keywords-from-a-url): TF-IDF, RAKE and YAKE vs model-based extraction, a competitor's page as a keyword brief, clustering, and the limits.
- [Open Graph image sizes](https://urlpipe.dev/guides/open-graph-image-sizes): The og:image size and file limits each network documents, checked against its own docs, and the one size that works everywhere.
- [Screenshot a website programmatically](https://urlpipe.dev/guides/screenshot-a-website-programmatically): Puppeteer vs Playwright vs an API, and the full-page gotchas: lazy loading, sticky headers, banners and size limits.
- [Core Web Vitals: lab vs field](https://urlpipe.dev/guides/core-web-vitals-lab-vs-field): LCP, INP and CLS, their thresholds, and why a Lighthouse run and real-user data tell different stories.
- [Find JavaScript errors on any site](https://urlpipe.dev/guides/find-javascript-errors-on-any-site): DevTools, headless Chrome and load-time captures — and what each can and cannot see on a site you don't own.
- [Verify webhook signatures (HMAC)](https://urlpipe.dev/guides/verify-webhook-signatures-hmac): The signed string, the raw body, timestamp tolerance, constant-time compares and rotation — with code in five languages.
- [Chunking web pages for RAG](https://urlpipe.dev/guides/chunking-web-pages-for-rag): Markdown first, heading-aware splitting, chunk size and overlap, metadata, and keeping chunks fresh.
- [Polite scraping: robots.txt and pacing](https://urlpipe.dev/guides/polite-scraping-robots-txt-and-rate-limits): What RFC 9309 says about robots.txt, how to pace requests per host, and how to back off when a site struggles.
- [Idempotency keys explained](https://urlpipe.dev/guides/idempotency-keys-explained): What the header does, how servers implement it, and the mistakes that make a retry charge twice.
- [What llms.txt is](https://urlpipe.dev/guides/what-is-llms-txt): The format section by section, a real example, and an honest look at which tools read it.
- [Cookie banners and ads in scraped pages](https://urlpipe.dev/guides/cookie-banners-and-ads-in-scraped-pages): Why automated visits get every banner and ad, and how to remove them without consenting on anyone's behalf.

## Product

- [URL to Markdown API](https://urlpipe.dev/url-to-markdown-api): Turn any web page into clean, LLM-ready Markdown — rendered in a real browser, converted without a model, measured against the strategies other APIs use.
- [MCP server](https://urlpipe.dev/mcp-server): A hosted MCP server with 14 tools: your agent reads rendered pages as Markdown, looks at screenshots and pulls metadata, with one bearer token and nothing to install.
- [Screenshot API](https://urlpipe.dev/screenshot-api): Full-page website screenshots as PNG, JPEG or WebP — any viewport, retina, one element, dark mode, banners removed — at one price whatever the options.
- [Lighthouse API](https://urlpipe.dev/lighthouse-api): Run Google Lighthouse 13 on any URL by API — scores, Core Web Vitals lab metrics and every audit — and combine it with a screenshot and console errors.
- [Metadata API](https://urlpipe.dev/metadata-api): One clean JSON object per URL — title, description, image, favicon, author, date, feed — reconciled from Open Graph, Twitter cards, JSON-LD and HTML.
- [Console API](https://urlpipe.dev/console-api): Load any URL in a real browser and get back the JavaScript errors, warnings, uncaught exceptions and unhandled rejections it produced — no SDK, no access to the site.
- [Rendered HTML API](https://urlpipe.dev/rendered-html-api): Get the fully rendered HTML of any URL — scripts run, redirects followed, late content waited for — from a real browser, for the price of one page fetch.

## Use cases

- [Web access for AI agents](https://urlpipe.dev/use-cases/ai-agents): Give an agent a tool that reads any page — JavaScript rendered — as Markdown, sees it as a screenshot, and checks it for errors.
- [Web pages into a RAG pipeline](https://urlpipe.dev/use-cases/rag-ingestion): Turn a list of URLs into clean Markdown chunks with titles and dates — deterministic, measured, and free to re-fetch inside your freshness window.
- [Link previews and unfurls](https://urlpipe.dev/use-cases/link-previews): Build the card a pasted link turns into: title, description and image from /meta, and a screenshot when the page has no image of its own.
- [Monitoring client websites](https://urlpipe.dev/use-cases/client-site-monitoring): For agencies: JavaScript errors, Lighthouse scores and screenshots for every client site, tagged per client so usage splits cleanly for billing.
- [Monitoring pages for changes](https://urlpipe.dev/use-cases/content-monitoring): Detect when a web page's content changes: fetch it fresh as deterministic Markdown, hash it, and diff only when the hash moves.
- [Archiving pages as screenshots](https://urlpipe.dev/use-cases/visual-archiving): Keep a visual record of pages over time: full-page WebP captures, light and dark, with the rendered HTML beside each one.
- [URL features inside a multi-tenant app](https://urlpipe.dev/use-cases/multi-tenant-apps): Build link previews, imports and page reading into your own SaaS: one key per environment, a label per tenant, retries that never bill twice, EU processing.

## Integrations

- [Claude Desktop](https://urlpipe.dev/integrations/claude-desktop): Add URLpipe to Claude Desktop as a custom connector, or through a local bridge, so Claude can read, screenshot and audit any page.
- [Claude Code](https://urlpipe.dev/integrations/claude-code): One command connects Claude Code to URLpipe, so it can read docs pages, screenshot a deployed preview and check it for console errors.
- [Cursor](https://urlpipe.dev/integrations/cursor): Give Cursor's agent a rendered-page reader: add URLpipe to mcp.json and it reads docs, SPAs and preview deploys as Markdown.
- [Windsurf](https://urlpipe.dev/integrations/windsurf): Add URLpipe to Windsurf's mcp_config.json with serverUrl and a header, and Cascade can read, screenshot and audit any page.
- [VS Code](https://urlpipe.dev/integrations/vs-code): Add URLpipe to VS Code's mcp.json with a password prompt for the token, and Copilot's agent mode can read and audit any page.
- [Cline](https://urlpipe.dev/integrations/cline): Connect Cline to URLpipe with a streamableHttp entry in cline_mcp_settings.json, and it can read, render and screenshot any page.
- [Zed](https://urlpipe.dev/integrations/zed): Add URLpipe to Zed's context_servers with a url and an Authorization header, and the Agent Panel can read and inspect any page.
- [n8n](https://urlpipe.dev/integrations/n8n): Call URLpipe from n8n's HTTP Request node — an importable workflow for sync calls, and a Webhook node for async results.
- [Make](https://urlpipe.dev/integrations/make): Call URLpipe from Make's HTTP "Make a request" module, and receive async results with a Custom webhook scenario.
- [Zapier](https://urlpipe.dev/integrations/zapier): Call URLpipe from a Zap with Webhooks by Zapier's Custom Request, and catch async results with a Catch Hook.
- [Pipedream](https://urlpipe.dev/integrations/pipedream): Call URLpipe from a Pipedream Node.js step with fetch, and use an HTTP trigger's endpoint as the webhook for async results.
- [Google Sheets](https://urlpipe.dev/integrations/google-sheets): Pull titles, descriptions and Lighthouse scores into a spreadsheet with a short Apps Script and UrlFetchApp.
- [Airtable](https://urlpipe.dev/integrations/airtable): Enrich Airtable records with page metadata and screenshot links from an automation's Run a script action.
- [LangChain](https://urlpipe.dev/integrations/langchain): A 20-line LangChain document loader over /markdown: rendered pages in, Documents with source metadata out.
- [LlamaIndex](https://urlpipe.dev/integrations/llamaindex): A short LlamaIndex reader over /markdown: rendered pages in, Documents with their source URL out, ready for an index.

## Code recipes

- [Code recipes in Python](https://urlpipe.dev/code/python): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in Node.js](https://urlpipe.dev/code/node): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in PHP](https://urlpipe.dev/code/php): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in Ruby](https://urlpipe.dev/code/ruby): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in Go](https://urlpipe.dev/code/go): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in Java](https://urlpipe.dev/code/java): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in C#](https://urlpipe.dev/code/csharp): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in cURL](https://urlpipe.dev/code/curl): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.

## Glossary

- [Rendered DOM](https://urlpipe.dev/glossary/rendered-dom): The page as it exists after the browser has run its JavaScript — what a visitor sees, and what curl never returns.
- [Client-side rendering](https://urlpipe.dev/glossary/client-side-rendering): When the browser, not the server, builds the page from JavaScript — which is why curl gets an empty shell.
- [Hydration](https://urlpipe.dev/glossary/hydration): Attaching JavaScript to HTML the server already rendered, so a static page becomes an interactive app.
- [Headless browser](https://urlpipe.dev/glossary/headless-browser): A real browser engine run without a window, driven by code — how you read, screenshot and measure pages at scale.
- [robots.txt](https://urlpipe.dev/glossary/robots-txt): The file at a site's root that tells automated clients which paths they may fetch — a request, not a lock.
- [networkidle](https://urlpipe.dev/glossary/networkidle): The wait condition that treats a page as loaded once its network traffic goes quiet.
- [DOMContentLoaded](https://urlpipe.dev/glossary/domcontentloaded): The event fired once the HTML is parsed and deferred scripts have run — before images, and often before content.
- [Residential proxy](https://urlpipe.dev/glossary/residential-proxy): Sending a request out through a home internet connection's IP address instead of a datacentre's.
- [Bot detection](https://urlpipe.dev/glossary/bot-detection): How sites decide whether a visitor is a person or a program — and what they do about it.
- [Webhook signature](https://urlpipe.dev/glossary/webhook-signature): A keyed hash sent with each webhook so the receiver can prove who sent it and that nothing changed.
- [Idempotency key](https://urlpipe.dev/glossary/idempotency-key): A unique value sent with a request so retrying it can never do the work — or the charge — twice.
- [Cache TTL](https://urlpipe.dev/glossary/cache-ttl): How long a stored result stays fresh enough to reuse — the knob between cost and freshness.
- [SSRF](https://urlpipe.dev/glossary/ssrf): Tricking a server into fetching a URL on its own internal network — the first risk of any 'fetch this URL' feature.
- [Open Graph](https://urlpipe.dev/glossary/open-graph): The og: meta tags that decide the title, description and image a link preview shows.
- [og:image](https://urlpipe.dev/glossary/og-image): The one tag that decides the picture on every shared link to your page.
- [Twitter Cards](https://urlpipe.dev/glossary/twitter-cards): The twitter: meta tags that shape how a link looks on X — and how they fall back to Open Graph.
- [JSON-LD](https://urlpipe.dev/glossary/json-ld): Structured data in a script tag — the schema.org facts search engines read for rich results.
- [Full-page screenshot](https://urlpipe.dev/glossary/full-page-screenshot): An image of the whole page, top to bottom — not just the part that fits in the window.
- [Device pixel ratio](https://urlpipe.dev/glossary/device-pixel-ratio): How many physical pixels a screen uses for one CSS pixel — why retina screenshots are twice the size.
- [Cookie consent banner](https://urlpipe.dev/glossary/cookie-consent-banner): The consent pop-up European privacy law puts on most sites — and in most screenshots and scraped text.
- [Core Web Vitals](https://urlpipe.dev/glossary/core-web-vitals): Google's three user-experience metrics — loading, responsiveness and visual stability — and their thresholds.
- [Largest Contentful Paint (LCP)](https://urlpipe.dev/glossary/largest-contentful-paint): When the biggest image or text block in the viewport finishes rendering — the loading Core Web Vital.
- [Cumulative Layout Shift (CLS)](https://urlpipe.dev/glossary/cumulative-layout-shift): How much the page's content jumps around unexpectedly while you read it — the visual-stability Core Web Vital.
- [Interaction to Next Paint (INP)](https://urlpipe.dev/glossary/interaction-to-next-paint): How long the page takes to visibly respond to a click, tap or key press — the responsiveness Core Web Vital.
- [Total Blocking Time (TBT)](https://urlpipe.dev/glossary/total-blocking-time): How long long JavaScript tasks block the main thread during load — Lighthouse's heaviest-weighted metric.
- [Speed Index](https://urlpipe.dev/glossary/speed-index): How quickly the visible part of the page fills in during load, averaged over time.
- [Time to First Byte (TTFB)](https://urlpipe.dev/glossary/time-to-first-byte): How long before the server's response starts arriving — the floor under every other loading metric.
- [Unhandled promise rejection](https://urlpipe.dev/glossary/unhandled-promise-rejection): A JavaScript promise that failed with nothing there to catch it — the error that breaks features silently.
- [Keyword extraction](https://urlpipe.dev/glossary/keyword-extraction): Pulling the terms and phrases that say what a page is about out of its text — from any URL.
- [Markdown for LLMs](https://urlpipe.dev/glossary/markdown-for-llms): Converting a web page to Markdown before a model reads it — fewer tokens, same structure.
- [Context window](https://urlpipe.dev/glossary/context-window): How much text a language model can take in at once — the budget every fetched page spends from.
- [Chunking](https://urlpipe.dev/glossary/chunking): Splitting documents into pieces small enough to embed and retrieve — best done along the headings.
- [Embeddings](https://urlpipe.dev/glossary/embeddings): Lists of numbers that place text by meaning, so similar passages can be found without matching words.
- [llms.txt](https://urlpipe.dev/glossary/llms-txt): A Markdown file at a site's root that tells language models what the site is and where its best content lives.
- [Model Context Protocol (MCP)](https://urlpipe.dev/glossary/model-context-protocol): The open protocol that lets AI assistants and agents call external tools — like fetching a web page.
- [Streamable HTTP](https://urlpipe.dev/glossary/streamable-http): The transport remote MCP servers speak: JSON-RPC over plain HTTP POST, with streaming when it's needed.

## Compare

- [URLpipe vs Firecrawl](https://urlpipe.dev/compare/urlpipe-vs-firecrawl): Firecrawl crawls whole sites; URLpipe answers several questions about one URL. Where each wins.
- [URLpipe vs Jina Reader](https://urlpipe.dev/compare/urlpipe-vs-jina-reader): Both turn a URL into Markdown. Jina bills tokens; URLpipe charges a fixed price and does more per URL.
- [Best HTML-to-Markdown APIs](https://urlpipe.dev/best-html-to-markdown-apis): Eight ways to turn a web page into Markdown for a model, with measured output quality and prices.
- [Jina Reader alternatives](https://urlpipe.dev/alternatives/jina-reader): Leaving r.jina.ai? What to look for, the options, measured Markdown quality and how the calls map.
- [Microlink alternatives](https://urlpipe.dev/alternatives/microlink): Looking past Microlink's metadata API? How the endpoints map, what each costs and when to stay.
- [ScreenshotOne alternatives](https://urlpipe.dev/alternatives/screenshotone): When a screenshot API alone isn't enough: options compared, pricing, and how a ScreenshotOne call maps.
- [Firecrawl alternatives](https://urlpipe.dev/alternatives/firecrawl): Leaving Firecrawl? Which alternative fits whether you crawl whole sites or read one URL at a time.
- [Urlbox alternatives](https://urlpipe.dev/alternatives/urlbox): Alternatives to Urlbox for screenshots with Markdown and metadata — with a free tier.
- [PageSpeed Insights API alternatives](https://urlpipe.dev/alternatives/pagespeed-insights-api): When Google's free Lighthouse API isn't enough: async runs, webhooks, and Lighthouse next to the page's errors.
- [ScrapingBee alternatives](https://urlpipe.dev/alternatives/scrapingbee): When you need clean data from a URL more than proxies: alternatives to ScrapingBee compared.
- [Apify alternatives](https://urlpipe.dev/alternatives/apify): When a scraper platform is more than you need: per-URL alternatives to Apify for Markdown and more.
- [OpenGraph.io alternatives](https://urlpipe.dev/alternatives/opengraph-io): Alternatives to OpenGraph.io for Open Graph data and link previews, compared on price and fields.
- [Iframely alternatives](https://urlpipe.dev/alternatives/iframely): When you need a page's data rather than its embed: alternatives to Iframely compared.
- [URLpipe vs Microlink](https://urlpipe.dev/compare/urlpipe-vs-microlink): Two APIs that do several things with one URL. Where Microlink's GET-and-cache model wins, and where URLpipe's does.
- [URLpipe vs ScreenshotOne](https://urlpipe.dev/compare/urlpipe-vs-screenshotone): A screenshot specialist against a per-URL data API: options, pricing and data protection compared.
- [Best screenshot APIs](https://urlpipe.dev/best-screenshot-apis): Six website screenshot APIs compared on options, full-page capture, pricing and what else they return.
- [Best link preview APIs](https://urlpipe.dev/best-link-preview-apis): Five link preview and Open Graph APIs compared on fields, price per lookup and what else they do.
- [Best MCP servers for web browsing](https://urlpipe.dev/best-mcp-servers-for-web-browsing): Eight MCP servers that let an AI agent read the web, compared on JavaScript, hosting and cost.
- [Best Lighthouse APIs](https://urlpipe.dev/best-lighthouse-apis): Five ways to run Lighthouse from code, from Google's free API to monitoring products, compared.

## Research

- [Web-to-Markdown, measured](https://urlpipe.dev/research/markdown-benchmark): Four HTML-to-Markdown strategies on the same 33 real pages: how much of the visible text each keeps, and how much of its output the reader never saw.
- [What stops a headless browser in 2026](https://urlpipe.dev/research/what-stops-a-headless-browser): 46 well-known, frequently protected sites, visited from a datacentre and a residential address: which refuse outright, which answer 200 with a block page, and what a residential exit changes.
- [Lighthouse variance: how much does one audit move?](https://urlpipe.dev/research/lighthouse-variance): Ten mobile Lighthouse audits of each of five public pages, back to back: the performance score moved by up to 61 points on the same page.

## Engineering notes

- [Why we built our own rendering engine](https://urlpipe.dev/notes/our-own-rendering-engine): One warm Chrome, a fresh browser context per visit, about 900 lines with no dependencies — and the measurements that decided it.
- [One page visit, several answers](https://urlpipe.dev/notes/one-visit-scrape): How /scrape reads several results off one page load and stays identical to calling each endpoint on its own.
- [Pacing a site that can't keep up](https://urlpipe.dev/notes/pacing-slow-sites): When a target site falls over under load, slow down and try again — without failing the request or billing the wait.
- [Why async is the default](https://urlpipe.dev/notes/async-by-default): Pages take seconds to minutes and sometimes much longer; a request that returns a token at once never loses the answer.
- [Billing only what worked](https://urlpipe.dev/notes/billing-only-what-worked): Failures, bot checks, duplicates, cache hits and waits are free — and the harder case, a 200 that is really a block page.
- [Request timing as marks, not spans](https://urlpipe.dev/notes/request-timing-as-marks): Store four moments per request and derive every duration from them — and publish exactly one number.

## Changelog

- [Each project chooses whether to follow robots.txt](https://urlpipe.dev/changelog/robots-txt-per-project): A project setting, on by default: a page the site's robots.txt disallows is not fetched, and the refusal costs nothing.
- [Idempotency-Key, and identical requests share one run](https://urlpipe.dev/changelog/idempotency-key-and-in-flight-sharing): A retried request returns the first one's result and charge; a duplicate that arrives mid-run waits for it for free.
- [Labels: tag requests with your own ids](https://urlpipe.dev/changelog/labels): Attach up to 16 key/value labels to any request, get them back everywhere, and see usage broken down by label.
- [page_options: clean up the page before anything is read](https://urlpipe.dev/changelog/page-options): Block ads, remove cookie banners and your own selectors, or wait for an element — for every result read off the page.
- [screenshot_options: viewport, format, element, dark mode](https://urlpipe.dev/changelog/screenshot-options): PNG, JPEG or WebP, viewport and scale, one element by selector, dark mode, hidden selectors and injected CSS — at one credit.
- [Paid plans, VAT, and paid plans are never cut off](https://urlpipe.dev/changelog/paid-plans-and-overage): Starter, Pro and Scale on sale through Stripe; past the allowance, paid plans carry on and extra credits are billed.
- [Screenshots capture the whole page](https://urlpipe.dev/changelog/full-page-screenshots): Full page by default, with lazy-loaded images loaded first, up to a 16,384 px tall capture.
- [Every screenshot has a link that needs no API key](https://urlpipe.dev/changelog/screenshot-links-without-a-key): X-Result-Url gives a signed link to the image, valid for as long as the result is stored — drop it straight into an image tag.
- [Keep AI processing in the EU, per organization](https://urlpipe.dev/changelog/eu-ai-processing): An organization setting that keeps /meta, /summarize and /keywords inside the European Union, at no extra charge.
- [API keys are stored hashed, and shown once](https://urlpipe.dev/changelog/api-keys-shown-once): Project keys are stored as a digest, displayed only when minted, and can be rotated with a 24-hour overlap.
- [The URLpipe API as an MCP server](https://urlpipe.dev/changelog/mcp-server): Point Claude, Cursor or any MCP client at urlpipe.dev/mcp: 14 tools, the same arguments, results and credits as the API.
- [Fetch a page from a residential address](https://urlpipe.dev/changelog/residential-exits): residential: true routes the page visit through a home internet connection, for sites that answer datacentre traffic differently.
- [Parallel requests per plan](https://urlpipe.dev/changelog/parallel-requests-per-plan): Each plan runs a set number of requests at once — 1, 3, 8 or 20 — and says so on every response.
- [One credit meter, priced by what each call costs](https://urlpipe.dev/changelog/credits): The two separate AI and web allowances became one monthly credit balance, with each endpoint priced by what it costs to run.
- [/markdown converts without a language model](https://urlpipe.dev/changelog/deterministic-markdown): A DOM walk replaces the model: better coverage, half the stray text, about 20 ms a page, and no model cost.
- [Close your account or delete an organization yourself](https://urlpipe.dev/changelog/account-and-organization-deletion): Account settings closes an account; Organization settings deletes an organization and everything in it.
- [A site that can't keep up is paced, not failed](https://urlpipe.dev/changelog/pacing-slow-sites): When a target site slows down or answers 429/5xx, requests to it wait and retry instead of failing.
- [Filter requests and webhooks in the dashboard](https://urlpipe.dev/changelog/dashboard-filters): Narrow History by duration, URL and outcome, and the Webhooks table by endpoint and delivery time.
- [Send a failed webhook again by hand](https://urlpipe.dev/changelog/resend-a-failed-webhook): A Retry now button on any failed delivery, at any age, so a fixed endpoint can still receive the result.
- [Async requests no longer need a webhook](https://urlpipe.dev/changelog/results-without-a-webhook): report_to is optional: collect async results with GET /result/:token, or set a default webhook per project.
- [Signed webhooks](https://urlpipe.dev/changelog/signed-webhooks): Every delivery can carry an HMAC-SHA256 signature over the timestamp and raw body, with a 24-hour overlap on rotation.
- [Cache, timing and quota on every response](https://urlpipe.dev/changelog/response-headers): X-Result-Token, X-Cache, X-Cache-Age, X-Processing-Time-Ms and the quota headers on every response, and a meta object in webhooks.
- [POST /scrape: several results off one page visit](https://urlpipe.dev/changelog/scrape-endpoint): Ask for any combination of operations on one URL; the page loads once and every result comes back together.
- [Failed requests are never charged](https://urlpipe.dev/changelog/failed-requests-are-free): A timeout, a DNS error or the target's own 4xx/5xx spends nothing — and an error page is no longer returned as a result.
- [Free tools: try every endpoint without signing up](https://urlpipe.dev/changelog/free-tools): A no-signup page for each endpoint that runs it live on a URL you choose and shows the curl that reproduces it.
- [GET /result/:token](https://urlpipe.dev/changelog/get-result-by-token): Fetch any result by its token for 30 days — the way to collect an async result, or a sync one that timed out.
- [Per-project rate limits](https://urlpipe.dev/changelog/per-project-rate-limits): 60 requests a minute and 15 per 10 seconds per project key, with a 429 and Retry-After when you go over.
- [Sign in with Google or GitHub](https://urlpipe.dev/changelog/google-and-github-sign-in): Create an account or sign in with a Google or GitHub account instead of a password.
- [Public API documentation at /docs](https://urlpipe.dev/changelog/api-docs): A docs site with a quickstart, the core concepts and a page per endpoint, open without an account.
- [Request caching with max_age](https://urlpipe.dev/changelog/request-caching): Repeat requests are served from stored results inside a freshness window you choose — free.

## Key pages

- [Homepage](https://urlpipe.dev/)
- [Pricing](https://urlpipe.dev/pricing)
- [Status](https://urlpipe.dev/status): Measured latency and success rate per endpoint.
- [About](https://urlpipe.dev/about)
- [Security](https://urlpipe.dev/security)
- [Terms](https://urlpipe.dev/terms)
- [Privacy](https://urlpipe.dev/privacy)
- [Full documentation as one file](https://urlpipe.dev/llms-full.txt)
- [OpenAPI description](https://urlpipe.dev/openapi.json)

## Contact

- Email: contact@urlpipe.dev
- Provided by Aliat Partner S.L., Barcelona, Spain