Integrations

URLpipe as a LangChain loader

Rendered pages as LangChain Documents, in about twenty lines you own.

By Roger Campos · Last updated: September 2026

TL;DR

A LangChain document loader for URLpipe is a BaseLoader subclass whose lazy_load posts each URL to https://urlpipe.dev/markdown and yields a Document with the Markdown as page_content and the URL as metadata. It is a recipe you paste into your project, not an official package, and it works with any splitter or vector store.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The idea

Why a custom loader

LangChain's built-in web loaders fetch HTML with a plain HTTP request, or drive a browser you run yourself. URLpipe renders the page in real Chrome on its side and returns Markdown — headings, lists, tables and code blocks kept, navigation and cookie banners dropped — so a MarkdownHeaderTextSplitter has real structure to split on. The conversion uses no model, so the same page gives the same Document every time.

A recipe, not a package

There is no official URLpipe package for LangChain. The class below is about twenty lines you copy into your code and change as you like; it depends only on langchain-core and requests.

Code

The loader

urlpipe_loader.py

import os
from typing import Iterator

import requests
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document


class URLpipeLoader(BaseLoader):
    """Load web pages as Markdown Documents through URLpipe's /markdown endpoint."""

    def __init__(self, urls: list[str], api_key: str | None = None, max_age: str | int = "7 days"):
        self.urls = urls
        self.max_age = max_age
        self.headers = {"Authorization": f"Bearer {api_key or os.environ['URLPIPE_API_KEY']}"}

    def lazy_load(self) -> Iterator[Document]:
        for url in self.urls:
            res = requests.post("https://urlpipe.dev/markdown", headers=self.headers, timeout=75,
                                json={"url": url, "sync": True, "max_age": self.max_age})
            res.raise_for_status()
            yield Document(page_content=res.text,
                           metadata={"source": url, "cache": res.headers.get("X-Cache")})

load(), lazy_load() and load_and_split() all work, because BaseLoader builds them on lazy_load.

Code

Using it

Split on headings, then embed

from langchain_text_splitters import MarkdownHeaderTextSplitter

docs = URLpipeLoader([
    "https://docs.example.com/getting-started",
    "https://docs.example.com/api/authentication",
]).load()

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)
chunks = [
    chunk
    for doc in docs
    for chunk in splitter.split_text(doc.page_content)
]
# add doc.metadata["source"] to each chunk's metadata, then hand them to your vector store

For titles and publication dates in your metadata, post to /scrape with "operations": ["markdown", "meta"] and read operations.markdown.result and operations.meta.result from the JSON — 6 credits a page instead of 1.

Scale

Things to change for production

Slow pages. A sync call waits up to 60 seconds and then answers 504 with a token. Catch it and collect the result from GET /result/:token, as the agent recipe does.
Parallelism. Your plan runs a fixed number of requests at once — the X-Concurrency-Limit header says how many — so a thread pool of that size is the fastest safe loader. More returns 429.
Big batches. For thousands of pages, send them async with report_to and build Documents in the webhook — see RAG ingestion.
Freshness. Re-loading inside max_age is a free cache hit; pass max_age=0 to force a fresh fetch.

Endpoints

The endpoints on this page

/markdown API docs /scrape API docs

FAQ

Frequently asked questions

Is there an official URLpipe LangChain integration?

No. This page is a recipe: a short BaseLoader subclass you copy into your project. It needs only langchain-core and requests.

Does the loader handle JavaScript-rendered pages?

Yes. URLpipe loads every page in headless Chrome and lets its scripts run before converting it to Markdown.

What does each Document cost?

1 credit for a page fetched fresh, and nothing for a page served from the cache inside max_age. A failed fetch is not billed.

Can I use the loader with async LangChain code?

BaseLoader provides alazy_load and aload on top of lazy_load. For real concurrency, override alazy_load with an async HTTP client and cap it at your plan's parallel limit.

Keep reading

Firecrawl alternatives Idempotency-Key, and identical requests share one run Web pages into a RAG pipeline Apify alternatives LlamaIndex Why AI agents read empty pages All of integrations

Connect it in five minutes.

Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.

Start free Email the team