Integrations
URLpipe in LlamaIndex
Rendered pages as LlamaIndex Documents, ready for VectorStoreIndex.
By Roger Campos · Last updated: September 2026
TL;DR
llama-index-readers-urlpipe is the LlamaIndex reader for URLpipe: pip install llama-index-readers-urlpipe, then UrlpipeReader().load_data(urls). Each URL becomes one Document with the page as Markdown, a stable id per URL, and the URL in its metadata; aload_data fetches several pages at once.
Free plan, no credit card. 1,000 credits a month.
Setup
Install
pip install llama-index-readers-urlpipe
export URLPIPE_API_KEY=your_project_keyIt builds on the official Python client, installed with it, and needs Python 3.10 or later and llama-index-core 0.13 or 0.14.
Code
Build an index
An index is only as good as the text it is built from. URLpipe renders each page in real Chrome, then writes its main content as Markdown with the navigation and cookie banners left out, and with no model involved, so the same page gives the same text every time. The headings survive, so MarkdownNodeParser splits along the page's own sections:
from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownNodeParser
from llama_index.readers.urlpipe import UrlpipeReader
documents = UrlpipeReader().load_data([
"https://docs.example.com/getting-started",
"https://docs.example.com/api/auth",
])
index = VectorStoreIndex.from_documents(documents, transformations=[MarkdownNodeParser()])
print(index.as_query_engine().query("How do I authenticate?"))Every Document has the same id_ each time you load that URL, so an ingestion pipeline with a docstore skips pages it already has. Only url from the metadata goes to the embedding model and the LLM.
Async
Many pages at once
import asyncio
from llama_index.readers.urlpipe import UrlpipeReader
async def main(urls: list[str]) -> None:
reader = UrlpipeReader(max_concurrency=8, continue_on_failure=True)
documents = await reader.aload_data(urls)
print(len(documents), "pages loaded")
asyncio.run(main(["https://docs.example.com/a", "https://docs.example.com/b"]))- Set
max_concurrencyto your plan's parallel request limit (4 by default). operation="summarize"indexes an AI summary of each page instead of the full text (17 credits a page).continue_on_failure=Trueskips a page that can't be read; a wrong key or spent credits still raise.- Re-reading pages within
max_age(7 days by default) costs nothing.
Endpoints
The endpoints on this page
- Turn any URL into clean Markdown
Paste a link and get the page's main content as tidy Markdown — headings, lists, links and code kept, navigation and cookie banners stripped. The format LLMs and RAG pipelines work best with.
Try it free - Summarize any web page with AI
Paste a link and get a concise AI summary of the page's main content — the substance, without the navigation, ads and boilerplate.
Try it free
FAQ
Frequently asked questions
Yes: pip install llama-index-readers-urlpipe, then from llama_index.readers.urlpipe import UrlpipeReader. It is open source at github.com/URLpipe/llama-index-readers-urlpipe.
Markdown carries the page's structure in a fraction of the tokens, and MarkdownNodeParser splits it along the page's own headings.
Yes. Every page is rendered in real Chrome before it is converted.
1,000 credits for the first build, and nothing for pages re-read inside max_age. See the pricing page for plans.
Connect it in five minutes.
Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.