Guide

Rendered vs. raw HTML why your scraper gets nothing

You curl a page and get an almost-empty document — no article, no product, no data. The content is there in a browser, so what happened? This guide explains client-side rendering and how to scrape around it.

By Roger Campos · Last updated: July 2026

TL;DR

curl and "view source" return the raw HTML the server sends — which, on a single-page app, is a near-empty shell plus a JavaScript bundle. The real content only exists after the browser runs that JavaScript. To scrape it you need the rendered DOM: load the page in a real (headless) browser, let it execute, then read the result.

Start free See the API

Free plan, no credit card. 50 AI + 500 web operations / month.

The symptom

The page is right there — so why is the HTML empty?

You open a URL in your browser and see a full article. You curl the same URL and get a few lines of markup with none of the text. Both are correct — they're just looking at different moments in the page's life.

curl and "view source" show you the raw HTML the server sent: the document before any JavaScript has run. On a traditional server-rendered or static site that's the whole page. On a modern single-page app it's a nearly empty shell whose job is to load a JavaScript bundle that then builds the page in the browser.

So when your scraper "gets nothing," it usually got exactly what the server sent — a shell. The content you can see never existed in that response; it was created client-side, after load.

The mechanism

Raw HTML vs. the rendered DOM

	Raw HTML (curl / view-source)	Rendered DOM (headless browser)
JavaScript has run	No	Yes
Client-rendered content present	No	Yes
Redirects & lazy content resolved	Partial	Yes
Speed / cost	Fast, cheap	Slower, needs a browser
Works on single-page apps	No	Yes

The rendered DOM is what you see in DevTools' Elements panel: the document after the browser parsed the HTML, executed the JavaScript, applied DOM changes and followed redirects. That's the thing you actually want to scrape.

Diagnose

How to tell if a site needs rendering

1
Open the page, then view its raw source (Ctrl+U or curl the URL).
2
Search the source for a sentence you can see on the page.
3
If it's there, the site is server-rendered — a plain fetch is enough.
4
If the source is a small shell with a large script bundle and none of the visible text, the content is client-rendered — you need a real browser to get it.

Solve

Getting the rendered DOM

The fix is to load the page in a real, headless browser — the same engine as Chrome — let its JavaScript execute, and read the resulting HTML. That's what a rendered-HTML fetcher does: navigate, wait for the content, follow redirects, and return the DOM a visitor sees rather than the empty shell the server sent.

Use it to scrape single-page apps that ship an empty HTML shell.
Grab the post-JavaScript DOM for parsing, extraction or conversion to Markdown.
Debug the gap between a page's source and what it actually renders.

When it's still broken

The page renders — but content is missing

Sometimes even the rendered DOM is incomplete: a section never appears, or the data is blank. That almost always means the JavaScript that builds it failed — a script error, a blocked request, a failed API call.

The fastest way to find out is to capture the page's console output while it loads. Errors, warnings and uncaught exceptions point straight at the broken script or request, instead of leaving you guessing why a render came back half-empty.

FAQ

Frequently asked questions

How do I tell if a site is client-rendered?

Compare what you see in the browser with the page's raw source (Ctrl+U) or a curl of the URL. If the source is a small shell with a big <script> bundle and none of the visible text, the content is rendered client-side and you'll need a real browser to scrape it.

Why does curl work on some sites but not others?

Server-rendered and static sites send complete HTML, so curl gets everything. Single-page apps (React, Vue, Svelte, Angular) send a shell and build the page in the browser, so curl gets almost nothing. Many sites are a mix.

What does "the rendered DOM" mean?

The Document Object Model after the browser has parsed the HTML, run the JavaScript, applied changes and followed redirects — i.e. what you'd see in DevTools' Elements panel, not what "view source" shows.

How do console errors help when scraping?

If a rendered page is still missing content, the JavaScript that builds it probably failed. Capturing the console output surfaces the errors and failed network requests behind an incomplete render — often faster than guessing.

Try it yourself

Free tools for this

No signup — run these on a real page right now, then call the same endpoint from your code.

/html API docs /console API docs

Keep reading

HTML to Markdown for LLMs Reading a Lighthouse audit Extracting page metadata All guides

Put this into practice.

Every URLpipe endpoint has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 50 AI + 500 web operations a month, no card.

Start free Email the team