Guide

Chunking web pages for RAG

A web page is not a PDF: it has an outline, boilerplate on every page and content that changes. This guide covers how to split pages into chunks a retriever can actually use.

By Roger Campos · Last updated: September 2026

TL;DR

To chunk web pages for RAG, convert each page to clean Markdown first, then split on its headings, and only split long sections further by paragraph up to a size limit — a few hundred tokens is a common start. Prefix each chunk with the page title and heading path, keep the URL and fetch date as metadata, and re-chunk a page only when its content changes.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The problem

Why web pages need their own chunking strategy

Most chunking advice is written for PDFs and plain text. Web pages differ in three ways that matter: they have an explicit outline, they repeat the same boilerplate on every page, and they change.

The outline is a gift. A page's h1–h6 headings already divide it into sections an author meant to be read as units, and a chunker that respects them produces chunks that each answer one question. A chunker that counts characters cuts a paragraph in half and glues the end of one section to the start of the next.

The boilerplate is a trap. Navigation, footers, cookie notices and "related posts" appear on every page of a site. Embed them and every chunk from that site looks alike to the retriever — and the footer becomes the most retrievable text you own.

The change is a maintenance problem. A PDF is written once; a pricing page is edited next week. Chunks need to know where they came from and when, so they can be replaced.

Step 1

Convert to clean Markdown before you split anything

Every decision below is easier on Markdown than on HTML. The headings are # lines you can split on with a regular expression, tables and code blocks are delimited, and the chrome is gone — if the converter isolates the main content. That last part is the one to check: a conversion that keeps the navigation hands you the boilerplate problem in a new format.

Render first for client-side pages, or you'll chunk an empty shell. And prefer a deterministic conversion over asking a model: on the 33-page benchmark in the HTML to Markdown guide, the model-based conversion covered the least of the page and added the most text the reader never saw.

Step 2

Split on headings, then on paragraphs

The strategy that works well for most web content is hierarchical: split at headings first, and only split a section further when it's too long for your limit — at paragraph boundaries, never inside a table or a code block. Short neighbouring sections can be merged so you don't end up with a chunk that is a heading and one line.

Heading-aware chunking of a Markdown page

import re

HEADING = re.compile(r"^(#{1,6})\s+(.*)$")

def sections(markdown):
    """Yield (heading_path, text) for each section, tracking the heading hierarchy."""
    path, lines, in_code = [], [], False
    for line in markdown.splitlines():
        if line.startswith("```"):
            in_code = not in_code
        match = None if in_code else HEADING.match(line)
        if match:
            if lines:
                yield list(path), "\n".join(lines).strip()
            level = len(match.group(1))
            path = path[: level - 1] + [match.group(2).strip()]
            lines = []
        else:
            lines.append(line)
    if lines:
        yield list(path), "\n".join(lines).strip()

def chunks(markdown, page_title, max_words=300):
    for path, text in sections(markdown):
        if not text:
            continue
        header = " > ".join([page_title, *path])
        buffer = []
        for para in re.split(r"\n\s*\n", text):
            if buffer and len(" ".join(buffer + [para]).split()) > max_words:
                yield header, "\n\n".join(buffer)
                buffer = []
            buffer.append(para)
        if buffer:
            yield header, "\n\n".join(buffer)

This splits paragraphs on blank lines, and a fenced code block with blank lines in it could still be divided; for code-heavy pages, treat each fenced block as one indivisible paragraph. Word counts are a stand-in for tokens — use your embedding model's tokenizer if you need the limit to be exact.

Step 3

Give every chunk its context

A chunk that says "It costs $12 a month and includes five seats" is useless without knowing what "it" is. The cheapest fix is to prefix each chunk with the page title and its heading path before embedding — the header in the code above — so the vector carries the context the sentence leans on.

Store with each chunk	Why
Source URL, and the heading's anchor if it has one	Citations a user can click, and deleting a page's chunks together
Page title and heading path	Context for the embedding, and for the model reading the chunk
Fetched-at date	Answering "is this current?" and scheduling re-fetches
Hash of the page's Markdown	Detecting whether a re-fetched page changed at all
Language, section or product tags	Filtering before similarity search, which beats hoping similarity alone gets it right

Tuning

Chunk size and overlap: how to choose

There is no correct chunk size, and any single number you read — including one here — is a starting point. A few hundred tokens is a common default. Smaller chunks retrieve more precisely and lose context; larger ones keep context and dilute the embedding with several topics at once.

The only reliable way to choose is to measure: write twenty or thirty questions your users actually ask, note which page section answers each, and check how often retrieval returns it in the top few results at two or three chunk sizes. That takes an afternoon and beats any rule of thumb.

Overlap helps when you split inside a section — a sentence or two carried over keeps an idea from being cut in half. At heading boundaries it matters much less.
Tables should stay whole where possible, with their header row repeated if you must split one; a row without its headers is a list of unlabeled numbers.
Code blocks should never be split mid-block. A function cut in two retrieves as noise.
Very short pages — a glossary entry, a changelog item — are often best as a single chunk.

Keeping it fresh

Re-fetching and re-chunking pages that change

1
Re-fetch each URL on a schedule that fits how often it changes — daily for pricing and news, monthly for documentation, rarely for archives.
2
Hash the new Markdown. If it matches the stored hash, stop: nothing to re-embed.
3
If it changed, delete every chunk for that URL, then chunk and embed the new version. Deleting by URL is simpler and safer than diffing chunks, and it guarantees no stale fragment survives.
4
If the fetch fails or returns something suspiciously short, keep the old chunks and flag the page. A consent wall or an error page is not a new version of the content.

With URLpipe

The Markdown half of the pipeline

URLpipe's /markdown endpoint is step 1: it renders the page in real Chrome, isolates the main content and converts it deterministically — no model — for 1 credit a page. The same rendered page always gives the same Markdown, which is what makes the hash check in the refresh loop meaningful.

Re-running is cheap by design. Results are reused for 7 days by default and cache hits are free, so running your pipeline again over pages fetched this week costs nothing; ask for fresher content with max_age when a page needs it — see caching and freshness. The chunking, embedding and storage are yours: URLpipe hands you the Markdown and stops there. If you want a per-page summary to store beside the chunks, /summarize produces one, but it runs a model and costs 17 credits, so use it where the summary is worth that.

FAQ

Frequently asked questions

What chunk size should I use for RAG?

There is no universal answer; it depends on your embedding model and your questions. A few hundred tokens per chunk is a common starting point. Measure retrieval on real questions and adjust, rather than copying a number.

Should chunks overlap?

When you split inside a section, a small overlap — a sentence or two — keeps an idea from being cut in half. When you split on headings, overlap matters less, because each chunk already starts at a natural boundary.

Why convert HTML to Markdown before chunking?

Raw HTML chunks are mostly markup and repeated navigation, which wastes embedding tokens and makes every chunk from a site look alike. Markdown keeps the headings you want to split on and drops the rest.

How do I keep chunks up to date when pages change?

Store a hash of each page's Markdown. When you re-fetch, compare hashes and re-embed only the pages that changed; delete the old chunks for that URL first so stale text doesn't linger in the index.

Try it yourself

Free tools for this

No signup — run these on a real page right now, then call the same endpoint from your code.

/markdown API docs /caching API docs

Keep reading

HTML to Markdown for LLMs Get keywords from a URL What llms.txt is Reading a Lighthouse audit Rendered vs. raw HTML All guides

Put this into practice.

Each of the eight kinds of data URLpipe returns has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 1,000 credits a month, no card.

Start free Email the team