cURL · Code recipe
Convert a web page to Markdown in cURL
Turn any URL into Markdown you can hand to a model, in any shell with curl with curl. Every program on this page runs as it stands — each one was run against a stub of the API before it was published.
By Roger Campos · Last updated: September 2026
TL;DR
To convert a web page to Markdown in cURL, POST its URL to https://urlpipe.dev/markdown with sync set to true; the response body is the Markdown, read with curl -o. The page is rendered in real Chrome first, then converted by a deterministic walk over the DOM, not a model, for 1 credit a page.
Free plan, no credit card. 1,000 credits a month.
One POST to /markdown renders the page in real Chrome, then walks the rendered DOM and writes the main content as Markdown, with navigation, footers and boilerplate left out. No model is involved, so the same page gives the same Markdown every time, in about 20 ms after the page has loaded.
In cURL the response is plain text, read with curl -o — there is no JSON envelope to unwrap. Measured on 33 pages, the output covered 87.2% of the text a visitor sees, and 11.0% of it was text the visitor never saw; the guide to Markdown for LLMs has the comparison.
Setup
Before you start
curl is on every Mac and almost every Linux box. jq reads the token in the async script; install it from your package manager if jq --version finds nothing.
curl --version | head -1
jq --version # brew install jq / apt install jq
export URLPIPE_API_KEY="your_api_key"
The request
Print a page as Markdown
"sync": true keeps the request open until the result is ready. -w '%{http_code}' hands you the status and -o parks the body in a file, so an error message is printed, not decoded or saved as if it were the result. --max-time stops a page that never finishes loading from hanging the script. The body is the Markdown itself: cat it, or -o page.md to keep it.
#!/usr/bin/env bash
set -euo pipefail
response=$(mktemp)
trap 'rm -f "$response"' EXIT
# -w prints the status; -o keeps the body out of the way until the status is known.
status=$(curl -sS --max-time 90 -o "$response" -w '%{http_code}' \
https://urlpipe.dev/markdown \
-H "Authorization: Bearer $URLPIPE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "sync": true}')
if [ "$status" != 200 ]; then
echo "URLpipe answered $status: $(cat "$response")" >&2
exit 1
fi
# Plain text: pipe it into a file, a chunker or a prompt.
cat "$response"
echo
Run it: bash html_to_markdown.sh
Async
The async variant: a token, a webhook and a poll
Leave out sync and the answer is a token, straight away; jq -r .token pulls it out without quotes. The loop polls every two seconds and the case covers the four answers GET /result/:token can give.
#!/usr/bin/env bash
set -euo pipefail
response=$(mktemp)
trap 'rm -f "$response"' EXIT
API=https://urlpipe.dev
AUTH="Authorization: Bearer $URLPIPE_API_KEY"
# No "sync": the request is accepted at once and the work carries on without you.
status=$(curl -sS -o "$response" -w '%{http_code}' "$API/markdown" \
-H "$AUTH" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"report_to": "https://your-app.com/webhooks/urlpipe",
"labels": {"customer": "acme"}
}')
if [ "$status" != 200 ]; then
echo "URLpipe answered $status: $(cat "$response")" >&2
exit 1
fi
token=$(jq -r .token "$response")
echo "Accepted $token"
# The result is POSTed to report_to when it is ready. Polling by token is the
# other way to collect it: no endpoint needed, and a backup for the webhook.
for _ in $(seq 60); do
status=$(curl -sS -o "$response" -w '%{http_code}' "$API/result/$token" -H "$AUTH")
[ "$status" = 202 ] || break # 202 means still processing
sleep 2
done
case "$status" in
200) ;;
202) echo "Still processing after two minutes; try the token again later." >&2; exit 1 ;;
422) echo "The analysis failed: $(jq -r .error "$response")" >&2; exit 1 ;;
410) echo "The result is past the 30-day window; send the request again." >&2; exit 1 ;;
*) echo "URLpipe answered $status: $(cat "$response")" >&2; exit 1 ;;
esac
# Plain text: pipe it into a file, a chunker or a prompt.
cat "$response"
echo
Run it: bash html_to_markdown_async.sh
Errors
Handle errors and retries
-D writes the response headers to a file, which is where Retry-After is read from. A 401 answers in plain text, so it is decided on the status alone; jq reads the error code of everything else.
#!/usr/bin/env bash
set -euo pipefail
response=$(mktemp)
headers=$(mktemp)
trap 'rm -f "$response" "$headers"' EXIT
# urlpipe PATH JSON: POST a sync request, leaving the result in $response.
# Retries the two 429s that clear by themselves; fails with a message otherwise.
urlpipe() {
local attempt status code detail retry_after
for attempt in 0 1 2 3 4; do
status=$(curl -sS --max-time 90 -o "$response" -D "$headers" -w '%{http_code}' \
"https://urlpipe.dev$1" \
-H "Authorization: Bearer $URLPIPE_API_KEY" \
-H "Content-Type: application/json" \
-d "$2")
case "$status" in
200) return 0 ;;
401) echo "URLpipe: 401: the API key is missing or wrong. Check URLPIPE_API_KEY." >&2; return 1 ;;
esac
# A body that is not JSON leaves both empty.
code=$(jq -r '.error // ""' "$response" 2>/dev/null || true)
detail=$(jq -r 'if .message then "\(.error): \(.message)" else .error end' "$response" 2>/dev/null || true)
if [ "$status" = 429 ] && [ "$code" = rate_limited ]; then
# Sending too fast: Retry-After says how long the window has left.
retry_after=$(awk 'tolower($1) == "retry-after:" { print $2 + 0 }' "$headers")
sleep "${retry_after:-1}"
elif [ "$status" = 429 ] && [ "$code" = concurrency_limit ]; then
# Every parallel slot on your plan is busy with your own requests.
sleep $((2 ** attempt))
elif [ "$status" = 504 ]; then
# Still running on our side; the token collects it from GET /result/:token.
echo "URLpipe: 504 processing_timeout: collect it later with token $(jq -r .token "$response")" >&2
return 1
else
# 403 email_unverified, 422 (a bad parameter, or a page that would not load),
# 429 quota_exceeded: sending the same request again gets the same answer.
echo "URLpipe: $status: $detail" >&2
return 1
fi
done
echo "URLpipe: 429: still refused after 5 attempts" >&2
return 1
}
urlpipe /markdown '{"url": "https://example.com", "sync": true}' || exit 1
# Plain text: pipe it into a file, a chunker or a prompt.
cat "$response"
echo
Run it: bash html_to_markdown_errors.sh
Details
What to know about /markdown
- It reads no CSS, so text hidden only by a stylesheet can end up in the Markdown.
- Pages over 10 MB of HTML are refused with
The page is too big to be processed. page_options.remove_selectorsdrops elements before conversion when a site's chrome survives the boilerplate rules.- A repeat request inside
max_age(7 days by default) is served from the cache and costs nothing.
FAQ
Frequently asked questions
Does the Markdown conversion use an LLM?
Does it work on pages built with JavaScript?
How do I keep the Markdown for later?
Do I need an SDK to call URLpipe from cURL?
Get a key and run it.
Free plan, no card. Paste your key into URLPIPE_API_KEY and every program on this page runs as it is.