Guide

How to convert HTML to Markdown for LLMs & RAG

Feeding raw HTML to a language model wastes tokens and confuses the model. Clean Markdown fixes both. Here's why it matters, what "clean" actually means, and how to get it — even from JavaScript-heavy pages.

By Roger Campos · Last updated: September 2026

TL;DR

Convert web pages to Markdown before sending them to an LLM: it strips the navigation, ads and markup that waste tokens, and keeps the structure (headings, lists, links, code) that models and retrieval rely on. The catch is JavaScript — you have to render the page in a real browser first, or client-rendered sites come back empty.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The problem

Why raw HTML is the wrong input for a model

A web page is written for a browser, not a language model. Most of it is scaffolding — and that scaffolding is expensive.

When you fetch a page, the bytes you get back are mostly not content. Navigation menus, footers, cookie banners, ad slots, inline styles, analytics snippets and framework markup can easily make up 80–90% of an article's HTML. None of it means anything to a model, but all of it costs tokens.

That has two consequences. First, cost and context: you pay for every token and you fill the context window with noise, so fewer real pages fit. Second, quality: boilerplate dilutes the signal, and models can latch onto navigation text or repeated chrome instead of the actual content.

Token waste — raw HTML is several times larger than the equivalent Markdown.
Diluted signal — the model has to find the content inside the markup.
Worse chunks — splitting raw HTML for RAG buries content in tags and breaks structure.

The fix

What a clean Markdown conversion keeps — and drops

The goal is to reduce a page to its main content while preserving the structure that carries meaning. Markdown is the sweet spot: compact, plain-text, and semantic enough for a model to understand headings, lists, emphasis, links and code.

Keep

Headings (the document's outline)
Paragraphs, lists and tables
Links and their anchor text
Code blocks, inline code and quotes
The main article or body content

Drop

Navigation, menus and breadcrumbs
Headers, footers and sidebars
Cookie banners and consent modals
Ads, tracking and analytics scripts
Inline styles and framework markup

The evidence

How well do the common conversion strategies work?

We ran four strategies over the same 33 pages — 14 well-known sites and 19 pages taken from real production requests — and scored each against the text a visitor actually sees.

Two numbers matter. Coverage is how much of the page's visible text made it into the output: miss it, and your model can't answer from it. Leak is the share of the output that is text the visitor never saw — cookie-policy boilerplate, hidden menus, screen-reader-only labels, script-generated fragments. Leak is worse than it sounds: the model reads it as the page.

Strategy	Coverage	Leak	Cost per page	Time per page
Deterministic walk of the rendered DOM (URLpipe /markdown)	87.2%	11.0%	$0	~20 ms
Readability (the strategy Jina Reader runs)	88.2%	21.3%	—	—
Selector blocklist (Firecrawl's strategy)	85.8%	17.0%	—	—
Ask a language model to convert the HTML	79.0%	22.0%	$0.0025	39 s

Coverage is close across the three deterministic strategies; leak is where they differ. Readability keeps the densest block of text and everything inside it, banner text included. A selector blocklist removes the elements it knows about and keeps whatever it doesn't. The lowest leak came from walking the rendered document and deciding, element by element, what is page and what is chrome.

One honest limitation applies to all four: none of them reads the page's CSS, so text hidden only by a stylesheet can still leak into the output. The three deterministic strategies cost nothing per page and run in milliseconds on hardware you already have; the model is the only one with a bill.

The argument

Why not let a model do the conversion?

It is tempting: models read HTML well, and "convert this page to Markdown" is a one-line prompt. Measured, it is the worst option on every axis. It covered the least text, leaked the most, cost money on every page and took the better part of a minute. On one long technical article the model spent 143 seconds echoing the input HTML back unchanged — and reported success.

The deeper problem is that a model's output can't be checked cheaply. A deterministic converter does the same thing to the same page every time: when it drops a table you can find out why and fix it once. A model drops a paragraph on one run and paraphrases a sentence on the next, and the only way to notice is to read both. For input you'll feed to another model, you want the step before it to be boring.

Use code to convert. The structure is already in the DOM; walking it is exact and free.
Use a model for judgment. Summaries, keywords and extraction are where a model earns its cost — on clean Markdown, not on raw HTML.
Keep the faithful copy. Store the Markdown, and summarize or extract from it as a separate step.

The catch

JavaScript: why a naive fetch comes back empty

Here's the trap that breaks most homemade converters. Fetch the raw HTML of a modern site — a docs page, a news app, anything built with React, Vue or Svelte — and you often get a near-empty shell. The content isn't in the HTML the server sends; it's added by JavaScript after the page loads in a browser.

So the order matters: you have to render the page in a real browser first, let its JavaScript run, and only then extract and convert. Skip that step and you'll silently convert an empty page — the worst kind of bug, because it looks like it worked.

This is the single most common reason web scraping "randomly" fails. The full explanation — how to tell if a site is client-rendered and how to get the real DOM — is in the guide on rendered vs. raw HTML. AI agents hit the same wall; we measured it in why your agent reads empty pages.

How to do it

A reliable pipeline, step by step

1
Render in a real browser. Load the URL in headless Chrome, follow redirects, and wait for the content to appear — so client-rendered pages produce complete HTML.
2
Isolate the main content. Strip navigation, headers, footers, sidebars and banners, keeping the article or primary body.
3
Convert to Markdown. Map headings, lists, links, tables and code to their Markdown equivalents, preserving structure.
4
Chunk on structure. For RAG, split on headings and paragraphs — not arbitrary character counts — so each chunk is a coherent unit. The chunking guide goes into sizes and metadata.
5
Cache aggressively. The same URL rarely changes minute to minute; caching results avoids re-rendering and re-billing the same page.

Watch out

Common pitfalls

Tables and code are where lazy converters fall apart — verify they survive intact.
Images usually become links or alt text; decide whether your pipeline needs them.
Paywalls and logins only expose what's publicly rendered — you can't extract what you can't see.
Over-summarizing too early — keep a faithful Markdown copy; summarize as a separate step so you don't lose detail your retrieval might need.

FAQ

Frequently asked questions

Why not just send raw HTML to the model?

Raw HTML is mostly boilerplate — navigation, scripts, inline styles, tracking, cookie banners — that carries no meaning but consumes tokens and dilutes the signal. Markdown keeps the page's actual content and its structure in a fraction of the tokens, which improves both cost and answer quality.

Does Markdown really save tokens?

Yes, substantially. A typical article's HTML is several times larger than the equivalent Markdown once nav, scripts and styling are removed. Fewer tokens per page means more pages fit in a context window and each RAG chunk is denser with real content.

How do JavaScript-heavy sites affect the conversion?

If you fetch the raw HTML of a single-page app you often get an empty shell — the content is added by JavaScript after load. To convert it you must render the page in a real browser first, then extract. See the guide on rendered vs. raw HTML.

Can't I just ask an LLM to convert the HTML to Markdown?

You can, but it is the worst of the strategies we measured. On 33 pages a model-based conversion covered 79.0% of the visible text and 22.0% of what it wrote was text the reader never saw, at about $0.0025 and 39 seconds a page. A deterministic walk of the rendered DOM covered 87.2% with 11.0%, in milliseconds, for nothing.

Should I chunk before or after converting to Markdown?

After. Convert to clean Markdown first, then chunk on structural boundaries (headings, paragraphs). Chunking raw HTML tends to split tags and bury content in markup, which hurts retrieval.

Try it yourself

Free tools for this

No signup — run these on a real page right now, then call the same endpoint from your code.

/markdown API docs /summarize API docs

Keep reading

Get keywords from a URL Chunking web pages for RAG What llms.txt is Reading a Lighthouse audit Rendered vs. raw HTML All guides

Put this into practice.

Each of the eight kinds of data URLpipe returns has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 1,000 credits a month, no card.

Start free Email the team