Skip to main content
URLpipe
Sign inStart free

URLpipe

Integrations

URLpipe in LangChain

Give an agent pages it can actually read, and load them into your RAG pipeline as clean Markdown.

By · Last updated: October 2026

TL;DR

langchain-urlpipe is the LangChain integration for URLpipe: pip install langchain-urlpipe. It adds six tools — urlpipe_read_page (the page as Markdown after its JavaScript ran), urlpipe_page_metadata, urlpipe_extract_data, urlpipe_technologies, urlpipe_console_errors and urlpipe_lighthouse — a UrlpipeToolkit with all six, and UrlpipeLoader, a document loader with sync and async loading.

Start freeSee the API

Free plan, no credit card. 1,000 credits a month.

Setup

Install

Terminal
pip install langchain-urlpipe
export URLPIPE_API_KEY=your_project_key

It builds on the official Python client, installed with it, and needs Python 3.10 or later and langchain-core 1.x. The key is a project API key from your dashboard.

Agents

Tools for an agent

LangChain's own web tools fetch a page with a plain HTTP request, so a single-page app or a docs site built with JavaScript comes back as an empty shell. URLpipe renders the page in real Chrome first. Each tool has a description written for the model, so it knows when to call it:

ToolReturnsCredits
urlpipe_read_pageThe page's main content as Markdown1
urlpipe_page_metadataTitle, description, language, main image, author, publication date, feed1
urlpipe_extract_dataThe values the agent names, read off the page with AI: exactly its fields, null where the page gives none24
urlpipe_technologiesWhat the site is built with: CMS, frameworks, analytics, CDN, with versions1
urlpipe_console_errorsThe errors, warnings and uncaught exceptions the page logs1
urlpipe_lighthouseLighthouse scores (0–100) and key metrics, mobile or desktop2
agent.py
from langchain.agents import create_agent
from langchain_urlpipe import UrlpipeToolkit

agent = create_agent("anthropic:claude-sonnet-5", tools=UrlpipeToolkit().get_tools())

result = agent.invoke(
    {"messages": [{"role": "user", "content": "Why is https://example.com slow on mobile?"}]}
)
print(result["messages"][-1].content)

When a page can't be loaded, or the credits run out, the tool returns the API's explanation to the model as the tool result, so the agent can try another page or tell you why.

Code

A loader for RAG

UrlpipeLoader turns URLs into Documents, one per page, with the Markdown as page_content and source, token and cache in the metadata. Markdown keeps the page's headings, so you can split on them:

load_and_split.py
from langchain_text_splitters import MarkdownHeaderTextSplitter
from langchain_urlpipe import UrlpipeLoader

docs = UrlpipeLoader(
    ["https://docs.example.com/getting-started", "https://docs.example.com/api/auth"],
    page_options={"block_cookie_banners": True},
).load()

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "h1"), ("##", "h2")])
chunks = []
for doc in docs:
    for chunk in splitter.split_text(doc.page_content):
        chunk.metadata.update(doc.metadata)  # keep the source URL on every chunk
        chunks.append(chunk)
  • operation="html" or "summarize" loads the rendered HTML or an AI summary instead.
  • alazy_load() loads several pages at once, max_concurrency at a time; set it to your plan's parallel limit.
  • continue_on_failure=True skips a page that can't be read instead of raising.
  • Loading a page again within max_age (7 days by default) is served from storage and costs nothing.
langchain-urlpipe on GitHubon PyPI

Endpoints

The endpoints on this page

/markdown API docs/meta API docs/extract API docs/technologies API docs/console API docs/lighthouse API docs

FAQ

Frequently asked questions

Is there an official URLpipe LangChain integration?
Yes: pip install langchain-urlpipe. It has six tools, a toolkit and a document loader, and is open source at github.com/URLpipe/langchain-urlpipe.
Do the tools read pages that need JavaScript?
Yes. URLpipe loads every page in real Chrome and lets its scripts run before converting it to Markdown.
What does a tool call cost?
What the matching API call costs: 1 credit to read a page, 1 for metadata, 24 to extract data, 2 for a Lighthouse audit. A cached result and a failed call cost nothing.
Does the loader work with async LangChain code?
Yes. alazy_load and aload fetch several pages at once, up to max_concurrency, and keep the order of your URLs.

Keep reading

Official LangChain and LlamaIndex packagesn8nMCP: prompts, a pricing resource, structured results, and two renamed toolsPipedreamRequest timing as marks, not spansMCP serverAll of integrations

Connect it in five minutes.

Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.

Start freeEmail the team