Guide

How to convert HTML to Markdown for LLMs & RAG

Feeding raw HTML to a language model wastes tokens and confuses the model. Clean Markdown fixes both. Here's why it matters, what "clean" actually means, and how to get it — even from JavaScript-heavy pages.

By Roger Campos · Last updated: July 2026

TL;DR

Convert web pages to Markdown before sending them to an LLM: it strips the navigation, ads and markup that waste tokens, and keeps the structure (headings, lists, links, code) that models and retrieval rely on. The catch is JavaScript — you have to render the page in a real browser first, or client-rendered sites come back empty.

Start free See the API

Free plan, no credit card. 50 AI + 500 web operations / month.

The problem

Why raw HTML is the wrong input for a model

A web page is written for a browser, not a language model. Most of it is scaffolding — and that scaffolding is expensive.

When you fetch a page, the bytes you get back are mostly not content. Navigation menus, footers, cookie banners, ad slots, inline styles, analytics snippets and framework markup can easily make up 80–90% of an article's HTML. None of it means anything to a model, but all of it costs tokens.

That has two consequences. First, cost and context: you pay for every token and you fill the context window with noise, so fewer real pages fit. Second, quality: boilerplate dilutes the signal, and models can latch onto navigation text or repeated chrome instead of the actual content.

Token waste — raw HTML is several times larger than the equivalent Markdown.
Diluted signal — the model has to find the content inside the markup.
Worse chunks — splitting raw HTML for RAG buries content in tags and breaks structure.

The fix

What a clean Markdown conversion keeps — and drops

The goal is to reduce a page to its main content while preserving the structure that carries meaning. Markdown is the sweet spot: compact, plain-text, and semantic enough for a model to understand headings, lists, emphasis, links and code.

Keep

Headings (the document's outline)
Paragraphs, lists and tables
Links and their anchor text
Code blocks, inline code and quotes
The main article or body content

Drop

Navigation, menus and breadcrumbs
Headers, footers and sidebars
Cookie banners and consent modals
Ads, tracking and analytics scripts
Inline styles and framework markup

The catch

JavaScript: why a naive fetch comes back empty

Here's the trap that breaks most homemade converters. Fetch the raw HTML of a modern site — a docs page, a news app, anything built with React, Vue or Svelte — and you often get a near-empty shell. The content isn't in the HTML the server sends; it's added by JavaScript after the page loads in a browser.

So the order matters: you have to render the page in a real browser first, let its JavaScript run, and only then extract and convert. Skip that step and you'll silently convert an empty page — the worst kind of bug, because it looks like it worked.

This is the single most common reason web scraping "randomly" fails. The full explanation — how to tell if a site is client-rendered and how to get the real DOM — is in the guide on rendered vs. raw HTML.

How to do it

A reliable pipeline, step by step

1
Render in a real browser. Load the URL in headless Chrome, follow redirects, and wait for the content to appear — so client-rendered pages produce complete HTML.
2
Isolate the main content. Strip navigation, headers, footers, sidebars and banners, keeping the article or primary body.
3
Convert to Markdown. Map headings, lists, links, tables and code to their Markdown equivalents, preserving structure.
4
Chunk on structure. For RAG, split on headings and paragraphs — not arbitrary character counts — so each chunk is a coherent unit.
5
Cache aggressively. The same URL rarely changes minute to minute; caching results avoids re-rendering and re-billing the same page.

Watch out

Common pitfalls

Tables and code are where lazy converters fall apart — verify they survive intact.
Images usually become links or alt text; decide whether your pipeline needs them.
Paywalls and logins only expose what's publicly rendered — you can't extract what you can't see.
Over-summarizing too early — keep a faithful Markdown copy; summarize as a separate step so you don't lose detail your retrieval might need.

FAQ

Frequently asked questions

Why not just send raw HTML to the model?

Raw HTML is mostly boilerplate — navigation, scripts, inline styles, tracking, cookie banners — that carries no meaning but consumes tokens and dilutes the signal. Markdown keeps the page's actual content and its structure in a fraction of the tokens, which improves both cost and answer quality.

Does Markdown really save tokens?

Yes, substantially. A typical article's HTML is several times larger than the equivalent Markdown once nav, scripts and styling are removed. Fewer tokens per page means more pages fit in a context window and each RAG chunk is denser with real content.

How do JavaScript-heavy sites affect the conversion?

If you fetch the raw HTML of a single-page app you often get an empty shell — the content is added by JavaScript after load. To convert it you must render the page in a real browser first, then extract. See the guide on rendered vs. raw HTML.

Should I chunk before or after converting to Markdown?

After. Convert to clean Markdown first, then chunk on structural boundaries (headings, paragraphs). Chunking raw HTML tends to split tags and bury content in markup, which hurts retrieval.

Try it yourself

Free tools for this

No signup — run these on a real page right now, then call the same endpoint from your code.

/markdown API docs /summarize API docs

Keep reading

Reading a Lighthouse audit Rendered vs. raw HTML Extracting page metadata All guides

Put this into practice.

Every URLpipe endpoint has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 50 AI + 500 web operations a month, no card.

Start free Email the team