Ruby · Code recipe
Convert a web page to Markdown in Ruby
Turn any URL into Markdown you can hand to a model, in Ruby 3.1+ with Net::HTTP from the standard library. Every program on this page runs as it stands — each one was run against a stub of the API before it was published.
By Roger Campos · Last updated: September 2026
TL;DR
To convert a web page to Markdown in Ruby, POST its URL to https://urlpipe.dev/markdown with sync set to true; the response body is the Markdown, read with res.body. The page is rendered in real Chrome first, then converted by a deterministic walk over the DOM, not a model, for 1 credit a page.
Free plan, no credit card. 1,000 credits a month.
One POST to /markdown renders the page in real Chrome, then walks the rendered DOM and writes the main content as Markdown, with navigation, footers and boilerplate left out. No model is involved, so the same page gives the same Markdown every time, in about 20 ms after the page has loaded.
In Ruby the response is plain text, read with res.body — there is no JSON envelope to unwrap. Measured on 33 pages, the output covered 87.2% of the text a visitor sees, and 11.0% of it was text the visitor never saw; the guide to Markdown for LLMs has the comparison.
Setup
Before you start
Nothing to install: net/http and json are part of Ruby. Put your API key in the environment so it never lands in the file.
ruby --version # 3.1 or later
export URLPIPE_API_KEY="your_api_key"
The request
Print a page as Markdown
sync: true keeps the request open until the result is ready. use_ssl: has to be asked for — Net::HTTP does not infer it from the scheme — and abort prints to stderr and exits 1. The body is the Markdown itself: res.body is the whole job.
require "json"
require "net/http"
uri = URI("https://urlpipe.dev/markdown")
# read_timeout: a sync call can take up to 60 s, Net::HTTP's default.
res = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https", read_timeout: 90) do |http|
http.post(uri.path, { url: "https://example.com", sync: true }.to_json,
"Authorization" => "Bearer #{ENV.fetch("URLPIPE_API_KEY")}",
"Content-Type" => "application/json")
end
abort "URLpipe answered #{res.code}: #{res.body}" unless res.is_a?(Net::HTTPOK)
# Plain text: pipe it into a file, a chunker or a prompt.
puts res.body
Run it: ruby html_to_markdown.rb
Async
The async variant: a token, a webhook and a poll
Leave out sync and the answer is a token, straight away. Net::HTTP reports the status as a string, so the poll compares res.code with "202", not the number.
require "json"
require "net/http"
API = URI("https://urlpipe.dev")
def send_request(request)
request["Authorization"] = "Bearer #{ENV.fetch("URLPIPE_API_KEY")}"
Net::HTTP.start(API.host, API.port, use_ssl: API.scheme == "https") { |http| http.request(request) }
end
# No sync: the request is accepted at once and the work carries on without you.
post = Net::HTTP::Post.new(URI.join(API, "/markdown"), "Content-Type" => "application/json")
post.body = {
url: "https://example.com",
report_to: "https://your-app.com/webhooks/urlpipe",
labels: { customer: "acme" }
}.to_json
res = send_request(post)
abort "URLpipe answered #{res.code}: #{res.body}" unless res.is_a?(Net::HTTPOK)
token = JSON.parse(res.body).fetch("token")
puts "Accepted #{token}"
# The result is POSTed to report_to when it is ready. Polling by token is the
# other way to collect it: no endpoint needed, and a backup for the webhook.
60.times do
res = send_request(Net::HTTP::Get.new(URI.join(API, "/result/#{token}")))
break unless res.code == "202" # 202 means still processing
sleep 2
end
case res.code
when "200" then nil
when "202" then abort "Still processing after two minutes; try the token again later."
when "422" then abort "The analysis failed: #{JSON.parse(res.body)["error"]}"
when "410" then abort "The result is past the 30-day window; send the request again."
else abort "URLpipe answered #{res.code}: #{res.body}"
end
# Plain text: pipe it into a file, a chunker or a prompt.
puts res.body
Run it: ruby html_to_markdown_async.rb
Errors
Handle errors and retries
Pattern matching on [status, code] keeps the retry rules in one place. A 401 answers in plain text, so it is checked before JSON.parse, and a body that still is not JSON becomes an empty hash.
require "json"
require "net/http"
class URLpipeError < StandardError; end
# POST a sync request and return the response, or raise URLpipeError.
def urlpipe(path, payload, attempts: 5)
uri = URI("https://urlpipe.dev#{path}")
attempts.times do |attempt|
res = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https", read_timeout: 90) do |http|
http.post(uri.path, payload.merge(sync: true).to_json,
"Authorization" => "Bearer #{ENV.fetch("URLPIPE_API_KEY")}",
"Content-Type" => "application/json")
end
return res if res.code == "200"
raise URLpipeError, "401: the API key is missing or wrong. Check URLPIPE_API_KEY." if res.code == "401"
body = begin
JSON.parse(res.body)
rescue JSON::ParserError
{}
end
code = body["error"].to_s
detail = body["message"] ? "#{code}: #{body["message"]}" : code
case [ res.code, code ]
in [ "429", "rate_limited" ]
# Sending too fast: Retry-After says how long the window has left.
sleep Integer(res["Retry-After"] || 1)
in [ "429", "concurrency_limit" ]
# Every parallel slot on your plan is busy with your own requests.
sleep 2**attempt
in [ "504", _ ]
# Still running on our side; the token collects it from GET /result/:token.
raise URLpipeError, "504 processing_timeout: collect it later with token #{body["token"]}"
else
# 403 email_unverified, 422 (a bad parameter, or a page that would not load),
# 429 quota_exceeded: sending the same request again gets the same answer.
raise URLpipeError, "#{res.code}: #{detail}"
end
end
raise URLpipeError, "429: still refused after #{attempts} attempts"
end
begin
res = urlpipe("/markdown", { url: "https://example.com" })
rescue URLpipeError => e
abort "URLpipe: #{e.message}"
end
# Plain text: pipe it into a file, a chunker or a prompt.
puts res.body
Run it: ruby html_to_markdown_errors.rb
Details
What to know about /markdown
- It reads no CSS, so text hidden only by a stylesheet can end up in the Markdown.
- Pages over 10 MB of HTML are refused with
The page is too big to be processed. page_options.remove_selectorsdrops elements before conversion when a site's chrome survives the boilerplate rules.- A repeat request inside
max_age(7 days by default) is served from the cache and costs nothing.
FAQ
Frequently asked questions
Does the Markdown conversion use an LLM?
Does it work on pages built with JavaScript?
How do I keep the Markdown for later?
Do I need an SDK to call URLpipe from Ruby?
Get a key and run it.
Free plan, no card. Paste your key into URLPIPE_API_KEY and every program on this page runs as it is.