Integrations

URLpipe as a LlamaIndex reader

Rendered pages as LlamaIndex Documents — a reader you own, in about twenty lines.

By Roger Campos · Last updated: September 2026

TL;DR

A LlamaIndex reader for URLpipe is a BaseReader subclass whose load_data posts each URL to https://urlpipe.dev/markdown and returns Documents with the Markdown as text and the URL in metadata. It is a recipe to paste into your project, not an official package, and plugs into VectorStoreIndex.from_documents.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The idea

Why a custom reader

An index is only as good as the text it is built from. URLpipe renders each page in real Chrome, then walks the result to Markdown with navigation and chrome removed — no model involved, so the text is the same every run. Markdown keeps the headings, which LlamaIndex's MarkdownNodeParser turns into nodes that follow the page's own sections.

A recipe, not a package

There is no official URLpipe reader on LlamaHub. The class below is a starting point you own; it depends only on llama-index-core and requests.

Code

The reader

urlpipe_reader.py

import os

import requests
from llama_index.core import Document
from llama_index.core.readers.base import BaseReader


class URLpipeReader(BaseReader):
    """Read web pages as Markdown through URLpipe's /markdown endpoint."""

    def __init__(self, api_key: str | None = None, max_age: str | int = "7 days"):
        self.max_age = max_age
        self.headers = {"Authorization": f"Bearer {api_key or os.environ['URLPIPE_API_KEY']}"}

    def load_data(self, urls: list[str]) -> list[Document]:
        documents = []
        for url in urls:
            res = requests.post("https://urlpipe.dev/markdown", headers=self.headers, timeout=75,
                                json={"url": url, "sync": True, "max_age": self.max_age})
            res.raise_for_status()
            documents.append(Document(text=res.text, metadata={"source": url}))
        return documents

Code

Building an index

Index a handful of pages

from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownNodeParser

documents = URLpipeReader().load_data([
    "https://docs.example.com/getting-started",
    "https://docs.example.com/api/authentication",
])

index = VectorStoreIndex.from_documents(
    documents,
    transformations=[MarkdownNodeParser()],
)
print(index.as_query_engine().query("How do I authenticate?"))

Each node keeps the source URL from its Document's metadata, so answers can cite the page they came from.

Want the page's title, author and publication date on every node? Post to /scrape with "operations": ["markdown", "meta"] instead, and put operations.meta.result into the Document's metadata — one page visit, 6 credits.

Scale

Things to change for production

Slow pages. A sync call waits up to 60 seconds, then answers 504 with a token and keeps working; collect it from GET /result/:token.
Parallelism. Run as many requests at once as your plan allows — the X-Concurrency-Limit header tells you — and no more, or the extra ones return 429.
Thousands of pages. Send them async with report_to and insert Documents as webhooks arrive; the RAG ingestion page has the handler.
Refreshing an index. Re-reading pages inside max_age costs nothing; a lower max_age buys fresher text at 1 credit a page.

Endpoints

The endpoints on this page

/markdown API docs /scrape API docs

FAQ

Frequently asked questions

Is there a URLpipe reader on LlamaHub?

Not yet. This is a recipe: a short BaseReader subclass you paste into your project, depending only on llama-index-core and requests.

Why Markdown instead of HTML for LlamaIndex?

Markdown carries the page's structure in a fraction of the tokens, and MarkdownNodeParser splits it along the page's own headings.

Does it read pages that need JavaScript?

Yes. Every page is rendered in headless Chrome before it is converted.

What does it cost to build an index of 1,000 pages?

1,000 credits for the first build, and nothing for pages re-read inside max_age. See the pricing page for plans.

Keep reading

URLpipe vs Firecrawl Idempotency-Key, and identical requests share one run Apify alternatives Web pages into a RAG pipeline LangChain Why AI agents read empty pages All of integrations

Connect it in five minutes.

Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.

Start free Email the team