Skip to main content

Confirm

Are you sure?

URLpipe
Sign inStart free

URLpipe

Python · Code recipe

Convert a web page to Markdown in Python

Turn any URL into Markdown you can hand to a model, in Python 3.9+ with requests. Every program on this page runs as it stands — each one was run against a stub of the API before it was published.

By · Last updated: September 2026

TL;DR

To convert a web page to Markdown in Python, POST its URL to https://urlpipe.dev/markdown with sync set to true; the response body is the Markdown, read with res.text. The page is rendered in real Chrome first, then converted by a deterministic walk over the DOM, not a model, for 1 credit a page.

Start freeSee the API

Free plan, no credit card. 1,000 credits a month.

One POST to /markdown renders the page in real Chrome, then walks the rendered DOM and writes the main content as Markdown, with navigation, footers and boilerplate left out. No model is involved, so the same page gives the same Markdown every time, in about 20 ms after the page has loaded.

In Python the response is plain text, read with res.text — there is no JSON envelope to unwrap. Measured on 33 pages, the output covered 87.2% of the text a visitor sees, and 11.0% of it was text the visitor never saw; the guide to Markdown for LLMs has the comparison.

Setup

Before you start

One dependency, requests. Put your API key in the environment so it never lands in the file.

Terminal
python3 -m pip install requests
export URLPIPE_API_KEY="your_api_key"

The request

Print a page as Markdown

"sync": True holds the connection open until the result is ready and answers with it. Pass timeout= every time: requests has no default, and a page that never finishes loading would otherwise hang your process. The body is the Markdown itself, so res.text is the whole job.

html_to_markdown.py
import os

import requests

res = requests.post(
    "https://urlpipe.dev/markdown",
    headers={"Authorization": f"Bearer {os.environ['URLPIPE_API_KEY']}"},
    json={"url": "https://example.com", "sync": True},
    # requests waits forever unless told otherwise; a sync call can take up to 60 s.
    timeout=90,
)
res.raise_for_status()

# Plain text: pipe it into a file, a chunker or a prompt.
print(res.text)

Run it: python3 html_to_markdown.py

Async

The async variant: a token, a webhook and a poll

Leave out sync and the answer is a token, straight away. for … else is the idiom for "ran out of attempts": the else runs only when the loop never hit break.

html_to_markdown_async.py
import os
import time

import requests

API = "https://urlpipe.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['URLPIPE_API_KEY']}"}

# No "sync": the request is accepted at once and the work carries on without you.
res = requests.post(
    f"{API}/markdown",
    headers=HEADERS,
    json={
        "url": "https://example.com",
        "report_to": "https://your-app.com/webhooks/urlpipe",
        "labels": {"customer": "acme"},
    },
    timeout=30,
)
res.raise_for_status()
token = res.json()["token"]
print(f"Accepted {token}")

# The result is POSTed to report_to when it is ready. Polling by token is the
# other way to collect it: no endpoint needed, and a backup for the webhook.
for _ in range(60):
    res = requests.get(f"{API}/result/{token}", headers=HEADERS, timeout=30)
    if res.status_code != 202:  # 202 means still processing
        break
    time.sleep(2)
else:
    raise SystemExit("Still processing after two minutes; try the token again later.")

if res.status_code == 422:
    raise SystemExit(f"The analysis failed: {res.json()['error']}")
if res.status_code == 410:
    raise SystemExit("The result is past the 30-day window; send the request again.")
res.raise_for_status()

# Plain text: pipe it into a file, a chunker or a prompt.
print(res.text)

Run it: python3 html_to_markdown_async.py

Errors

Handle errors and retries

raise_for_status() is fine for a script; a service wants to tell the failures apart. Check the status before calling res.json() — a 401 body is not JSON. sys.exit(message) prints to stderr and exits 1.

html_to_markdown_errors.py
import os
import sys
import time

import requests

API_KEY = os.environ["URLPIPE_API_KEY"]


class URLpipeError(Exception):
    pass


def urlpipe(path, payload, attempts=5):
    """POST a sync request and return the response, or raise URLpipeError."""
    for attempt in range(attempts):
        res = requests.post(
            f"https://urlpipe.dev{path}",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={**payload, "sync": True},
            timeout=90,
        )
        if res.status_code == 200:
            return res
        if res.status_code == 401:
            raise URLpipeError("401: the API key is missing or wrong. Check URLPIPE_API_KEY.")

        try:
            body = res.json()
        except ValueError:
            body = {}
        code = body.get("error", "")
        detail = f"{code}: {body['message']}" if body.get("message") else code

        if res.status_code == 429 and code == "rate_limited":
            # Sending too fast: Retry-After says how long the window has left.
            time.sleep(int(res.headers.get("Retry-After", 1)))
        elif res.status_code == 429 and code == "concurrency_limit":
            # Every parallel slot on your plan is busy with your own requests.
            time.sleep(2**attempt)
        elif res.status_code == 504:
            # Still running on our side; the token collects it from GET /result/:token.
            raise URLpipeError(f"504 processing_timeout: collect it later with token {body['token']}")
        else:
            # 403 email_unverified, 422 (a bad parameter, or a page that would not load),
            # 429 quota_exceeded: sending the same request again gets the same answer.
            raise URLpipeError(f"{res.status_code}: {detail}")
    raise URLpipeError(f"429: still refused after {attempts} attempts")


try:
    res = urlpipe("/markdown", {"url": "https://example.com"})
except URLpipeError as error:
    sys.exit(f"URLpipe: {error}")
except requests.RequestException as error:
    sys.exit(f"Network error: {error}")

# Plain text: pipe it into a file, a chunker or a prompt.
print(res.text)

Run it: python3 html_to_markdown_errors.py

Details

What to know about /markdown

  • It reads no CSS, so text hidden only by a stylesheet can end up in the Markdown.
  • Pages over 10 MB of HTML are refused with The page is too big to be processed.
  • page_options.remove_selectors drops elements before conversion when a site's chrome survives the boilerplate rules.
  • A repeat request inside max_age (7 days by default) is served from the cache and costs nothing.

Endpoints

The endpoints on this page

/markdown API docs

Other languages

Convert a web page to Markdown in another language

Node.jsPHPRubyGoJavaC#cURL

More Python: every Python recipe

FAQ

Frequently asked questions

Does the Markdown conversion use an LLM?
No. It is a deterministic walk over the rendered DOM, so it is fast, costs 1 credit a page, and gives the same output for the same page. Only /meta, /summarize and /keywords call a model.
Does it work on pages built with JavaScript?
Yes. The page is loaded in real Chrome and its scripts run before the DOM is read, so React, Vue and other client-rendered pages convert like static ones.
How do I keep the Markdown for later?
Write the body to a .md file, or rely on the cache: the same URL within max_age (7 days by default) is answered from storage for free, and any result can be fetched again by its token for 30 days.
Do I need an SDK to call URLpipe from Python?
requests is the only dependency; there is no SDK to install, because the API is one POST per job.

Keep reading

n8nWeb-to-Markdown, measuredConvert a web page to Markdown in Node.jsURLpipe vs FirecrawlFirecrawl alternativesKeyword extractionAll of code recipes

Get a key and run it.

Free plan, no card. Paste your key into URLPIPE_API_KEY and every program on this page runs as it is.

Start freeEmail the team