Integrations

URLpipe in LlamaIndex

Rendered pages as LlamaIndex Documents, ready for VectorStoreIndex.

By Roger Campos · Last updated: September 2026

TL;DR

llama-index-readers-urlpipe is the LlamaIndex reader for URLpipe: pip install llama-index-readers-urlpipe, then UrlpipeReader().load_data(urls). Each URL becomes one Document with the page as Markdown, a stable id per URL, and the URL in its metadata; aload_data fetches several pages at once.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

Setup

Install

Terminal

pip install llama-index-readers-urlpipe
export URLPIPE_API_KEY=your_project_key

It builds on the official Python client, installed with it, and needs Python 3.10 or later and llama-index-core 0.13 or 0.14.

Code

Build an index

An index is only as good as the text it is built from. URLpipe renders each page in real Chrome, then writes its main content as Markdown with the navigation and cookie banners left out, and with no model involved, so the same page gives the same text every time. The headings survive, so MarkdownNodeParser splits along the page's own sections:

build_index.py

from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownNodeParser
from llama_index.readers.urlpipe import UrlpipeReader

documents = UrlpipeReader().load_data([
    "https://docs.example.com/getting-started",
    "https://docs.example.com/api/auth",
])

index = VectorStoreIndex.from_documents(documents, transformations=[MarkdownNodeParser()])
print(index.as_query_engine().query("How do I authenticate?"))

Every Document has the same id_ each time you load that URL, so an ingestion pipeline with a docstore skips pages it already has. Only url from the metadata goes to the embedding model and the LLM.

Async

Many pages at once

load_async.py

import asyncio
from llama_index.readers.urlpipe import UrlpipeReader

async def main(urls: list[str]) -> None:
    reader = UrlpipeReader(max_concurrency=8, continue_on_failure=True)
    documents = await reader.aload_data(urls)
    print(len(documents), "pages loaded")

asyncio.run(main(["https://docs.example.com/a", "https://docs.example.com/b"]))

Set max_concurrency to your plan's parallel request limit (4 by default).
operation="summarize" indexes an AI summary of each page instead of the full text (17 credits a page).
continue_on_failure=True skips a page that can't be read; a wrong key or spent credits still raise.
Re-reading pages within max_age (7 days by default) costs nothing.

llama-index-readers-urlpipe on GitHub on PyPI

Endpoints

The endpoints on this page

/markdown API docs /summarize API docs

FAQ

Frequently asked questions

Is there an official URLpipe reader for LlamaIndex?

Yes: pip install llama-index-readers-urlpipe, then from llama_index.readers.urlpipe import UrlpipeReader. It is open source at github.com/URLpipe/llama-index-readers-urlpipe.

Why Markdown instead of HTML for LlamaIndex?

Markdown carries the page's structure in a fraction of the tokens, and MarkdownNodeParser splits it along the page's own headings.

Does it read pages that need JavaScript?

Yes. Every page is rendered in real Chrome before it is converted.

What does it cost to build an index of 1,000 pages?

1,000 credits for the first build, and nothing for pages re-read inside max_age. See the pricing page for plans.

Keep reading

Context window Chunking web pages for RAG Request caching with max_age HTML to Markdown for LLMs A sign-in form or an error page is a failure, not a result Chunking All of integrations

Connect it in five minutes.

Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.

Start free Email the team