Use cases

Web pages into your vector store

Send the URLs you already have. Get back Markdown with the chrome removed, ready to chunk and embed.

By Roger Campos · Last updated: September 2026

TL;DR

For RAG ingestion, convert each page to Markdown after it has rendered, strip navigation and boilerplate, and keep the source URL and title beside every chunk. URLpipe's /markdown does the conversion without a model — a deterministic walk of the rendered page — so the same page always gives the same text, and it costs 1 credit a page.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The job

What goes wrong when you embed web pages

A retrieval pipeline is only as good as the text you put in it. Raw HTML wastes tokens on markup. A naive text extraction keeps the cookie banner, the footer and forty navigation links, and those land in your chunks and come back as retrieval noise. A JavaScript-rendered page gives you nothing at all unless something runs it.

/markdown loads the page in headless Chrome, lets its scripts run, and walks the rendered DOM to Markdown: headings, paragraphs, lists, tables, fenced code and links as absolute URLs. It uses no model, so a page converts in milliseconds once it has loaded, and converting it twice gives the same bytes — which makes change detection and deduplication trivial.

How clean is clean? Measured on 33 pages

We measured four strategies on the same 33 pages: how much of the text a visitor sees each one keeps (coverage), and how much of its output is text the visitor never saw (leak).

Strategy	Coverage of visible text	Output the reader never saw
URLpipe /markdown (DOM walk)	87.2%	11.0%
Readability (what Jina Reader runs)	88.2%	21.3%
Selector blocklist (Firecrawl's approach)	85.8%	17.0%
An LLM asked to convert the page	79.0%	22.0%

Coverage is close across the board; the leak is where they differ, and leaked text is what pollutes retrieval. The LLM conversion was also the slowest and the only one with a per-page bill: about $0.0025 and 39 seconds a page, against nothing and about 20 ms for the DOM walk.

What you call

The endpoints

/markdown — the page as Markdown. 1 credit.
/scrape — several results off one page visit. ["markdown", "meta"] adds a title, description, language, authors and publication date for your chunk metadata, for the sum of the two: 2 credits.
Async webhooks and labels — send a batch, get each result delivered with the ids you tagged it with.

If the page's first heading is title enough, skip meta and spend half as much per page.

Building the index in LangChain or LlamaIndex? langchain-urlpipe and llama-index-readers-urlpipe turn a list of URLs straight into Documents, one per page, with the URL and its cache status as metadata — see the LangChain and LlamaIndex pages.

Flow

A batch, end to end

1
Collect your URLs — a sitemap you parse, a list of help-centre articles, the links in a CMS.
2
Post each one async with report_to and labels naming the source and your document id. Keep no more requests in flight than your plan's parallel limit — the X-Concurrency-Limit header tells you — because a queued async request counts too.
3
Your webhook receives each result with its labels, verifies the signature, chunks the Markdown and embeds it.
4
Refresh on a schedule. A repeat inside max_age (seven days unless you say otherwise) is served from the cache and costs nothing; ask for max_age: "1 day" when a source changes daily.

Submit one page of the batch

import os, requests

API = "https://urlpipe.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['URLPIPE_API_KEY']}"}

def submit(doc_id: str, url: str) -> str:
    res = requests.post(f"{API}/scrape", headers=HEADERS, json={
        "url": url,
        "operations": ["markdown", "meta"],
        "report_to": "https://your-app.com/webhooks/urlpipe",
        "labels": {"source": "help-centre", "doc": doc_id},
        "page_options": {"block_cookie_banners": True},
    }, timeout=30)
    res.raise_for_status()
    return res.json()["token"]   # keep it: GET /result/:token works for 30 days

const API = "https://urlpipe.dev";

export async function submit(docId, url) {
  const res = await fetch(`${API}/scrape`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.URLPIPE_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url,
      operations: ["markdown", "meta"],
      report_to: "https://your-app.com/webhooks/urlpipe",
      labels: { source: "help-centre", doc: docId },
      page_options: { block_cookie_banners: true },
    }),
  });
  if (!res.ok) throw new Error(`URLpipe ${res.status}`);
  return (await res.json()).token; // GET /result/:token works for 30 days
}

curl -X POST https://urlpipe.dev/scrape \
  -H "Authorization: Bearer $URLPIPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://help.example.com/articles/refunds",
    "operations": ["markdown", "meta"],
    "report_to": "https://your-app.com/webhooks/urlpipe",
    "labels": {"source": "help-centre", "doc": "a-1042"},
    "page_options": {"block_cookie_banners": true}
  }'

Receive it, chunk it, embed it

@app.post("/webhooks/urlpipe")
def urlpipe_webhook():
    body = request.get_data()          # raw bytes: verify before parsing
    verify_signature(request.headers, body)   # see /docs/async#signature
    delivery = json.loads(body)

    if not delivery["success"]:
        log.warning("skipped %s: %s", delivery["labels"], delivery["error"])
        return "", 200

    ops = delivery["result"]["operations"]
    markdown = ops["markdown"]["result"]
    meta = ops["meta"]["result"] if ops["meta"]["success"] else {}

    for i, chunk in enumerate(split_on_headings(markdown, max_tokens=500)):
        store.upsert(
            id=f'{delivery["labels"]["doc"]}:{i}',
            text=chunk,
            metadata={"title": meta.get("title"),
                      "published": meta.get("published_at"),
                      "source": delivery["labels"]["source"]},
        )
    return "", 200

Key the handler on the delivery's token or your own document id: deliveries are retried on failure, so the same result can arrive twice. The signature check is on the async page, in Python, JavaScript and PHP.

Credits

What it costs

A 5,000-page knowledge base, at list price. The first month ingests every page with its metadata:

What	A month	Credits each	Credits
First ingest: Markdown + metadata per page	5,000	2	10,000

10,000 credits; the cheapest plan that covers it is Starter: $19 a month for 20,000 credits. After that, refreshing the Markdown weekly and leaving the metadata alone:

What	A month	Credits each	Credits
Weekly refresh, Markdown only (4 a month)	20,000	1	20,000

20,000 credits; the cheapest plan that covers it is Starter: $19 a month for 20,000 credits. Anything you re-request inside the freshness window — a retried job, a second pipeline reading the same page — is a cache hit and free, and pages that fail to load cost nothing.

Limits

What URLpipe does not do for RAG

Find your URLs. There is no crawl and no sitemap walk. You bring the list; for whole-site crawling, a crawler is the right tool.
Chunk, embed or store. You get Markdown and metadata. Splitting, embedding and the vector store stay yours.
Read CSS. The conversion walks the DOM and does not evaluate stylesheets, so text hidden only by CSS can leak into the output — that is most of the 11.0%.
Extract a schema. It returns the page, not fields you define across thousands of pages.
Read behind a login. Public pages only. Pages over 10 MB of HTML are refused with a 422, free.

Endpoints

The endpoints on this page

/markdown API docs /scrape API docs

FAQ

Frequently asked questions

Does /markdown use an LLM?

No. It walks the rendered DOM and writes Markdown directly, so it is deterministic, takes milliseconds after the page loads, and costs the same as fetching the HTML. Only /meta, /summarize and /keywords call a model.

How do I keep the source URL with every chunk?

Put your own ids in labels — up to 16 keys per request. They come back in the webhook, the X-Labels header and GET /result/:token, so the handler knows which document a result belongs to without a lookup table.

Is re-ingesting the same pages billed again?

Not inside max_age, which defaults to 7 days and goes up to 30. A cached result is returned instantly and spends no credits, and a duplicate that arrives while the first is still running waits for it, also free.

Can it convert JavaScript-rendered documentation sites?

Yes. Every page is loaded in headless Chrome and its scripts run before the conversion. If content arrives late, page_options.wait_for_selector holds the capture until an element you name exists.

Which should I use, /markdown or /summarize, for RAG?

/markdown. Retrieval needs the source text, not a model's summary of it, and Markdown is 17 times cheaper per page.

Keep reading

Firecrawl alternatives Idempotency-Key, and identical requests share one run Apify alternatives A sign-in form or an error page is a failure, not a result Why AI agents read empty pages LlamaIndex All of use cases

Build it on the free plan.

Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.

Start free Email the team