Any URL in, clean Markdown out

The request

One POST, one bearer token

curl -X POST https://urlpipe.dev/markdown \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching", "sync": true, "page_options": {"block_cookie_banners":true}}'

import requests

res = requests.post(
    "https://urlpipe.dev/markdown",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching", "sync": True, "page_options": {"block_cookie_banners": True}},
)

const res = await fetch("https://urlpipe.dev/markdown", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching", sync: true, page_options: {"block_cookie_banners":true} }),
})

$ch = curl_init("https://urlpipe.dev/markdown");
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => [
        "Authorization: Bearer YOUR_API_KEY",
        "Content-Type: application/json",
    ],
    CURLOPT_POSTFIELDS => json_encode(["url" => "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching", "sync" => true, "page_options" => ["block_cookie_banners" => true]]),
]);

$response = curl_exec($ch);

# HTTP caching

The HTTP cache stores a response associated with a request and reuses the stored response for subsequent requests.

There are several advantages to reusability. First, since there is no need to deliver the request to the origin server, then the closer the client and cache are, the faster the response will be. …

## [Types of caches](#types_of_caches)

In the [HTTP Caching](https://httpwg.org/specs/rfc9111.html) spec, there are two main types of caches: **private caches** and **shared caches**.

Choosing

How should I choose a URL-to-Markdown API for RAG?

1. 1
   It renders the page. A client-rendered site sends an empty <div id="root"> to anything that doesn't run its scripts. An API that fetches with plain HTTP returns that shell as a successful, near-empty document — and your pipeline embeds it. URLpipe loads every page in headless Chrome and converts the rendered DOM.
2. 2
   It keeps what a reader sees. Coverage of the visible text: an article that loses its second half to an over-eager boilerplate filter answers half the questions.
3. 3
   It doesn't add what a reader never saw. Collapsed menus, off-screen drawers, "skip to content" links, cookie-policy text: all of it is in the HTML, none of it is the page. In a vector store it becomes chunks that match queries they have no business matching. This is the number most tools never publish.
4. 4
   It is deterministic. The same page gives the same Markdown on every call, so a content hash tells you when a page actually changed, and you re-embed only then. A model-based conversion rewrites the output on every run.
5. 5
   Links survive. Relative links are resolved to absolute URLs, so a chunk can cite its source and an agent can follow a link without the page it came from.
6. 6
   The bill matches the work. Check whether re-fetching an unchanged page, or a page that failed, is charged. On URLpipe neither is: cache hits and failed requests cost nothing.

Tokens

How many tokens does Markdown save?

Limits

What it does not do

• It doesn't read CSS. The conversion walks the DOM, so text hidden only by a stylesheet can leak into the output. That is most of the 11.0% above. Elements marked hidden or aria-hidden are dropped.
• It doesn't crawl. One URL in, one document out. There is no link discovery, sitemap walking or whole-site job — send the URLs you already have.
• It doesn't convert PDFs. A URL that serves a PDF, an image or a download is refused as not a web page, and costs nothing.
• It doesn't extract to a schema. You get Markdown, not typed fields. For the page's title, author and date, use /meta.
• It has a size ceiling of 10 MB of HTML per page.