Guide

Rendered vs. raw HTML why your scraper gets nothing

You curl a page and get an almost-empty document — no article, no product, no data. The content is there in a browser, so what happened? This guide explains client-side rendering and how to scrape around it.

By Roger Campos · Last updated: September 2026

TL;DR

curl and "view source" return the raw HTML the server sends — which, on a single-page app, is a near-empty shell plus a JavaScript bundle. The real content only exists after the browser runs that JavaScript. To scrape it you need the rendered DOM: load the page in a real (headless) browser, let it execute, then read the result.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The symptom

The page is right there — so why is the HTML empty?

You open a URL in your browser and see a full article. You curl the same URL and get a few lines of markup with none of the text. Both are correct — they're just looking at different moments in the page's life.

curl and "view source" show you the raw HTML the server sent: the document before any JavaScript has run. On a traditional server-rendered or static site that's the whole page. On a modern single-page app it's a nearly empty shell whose job is to load a JavaScript bundle that then builds the page in the browser.

So when your scraper "gets nothing," it usually got exactly what the server sent — a shell. The content you can see never existed in that response; it was created client-side, after load.

The mechanism

Raw HTML vs. the rendered DOM

	Raw HTML (curl / view-source)	Rendered DOM (headless browser)
JavaScript has run	No	Yes
Client-rendered content present	No	Yes
Redirects & lazy content resolved	Partial	Yes
Speed / cost	Fast, cheap	Slower, needs a browser
Works on single-page apps	No	Yes

The rendered DOM is what you see in DevTools' Elements panel: the document after the browser parsed the HTML, executed the JavaScript, applied DOM changes and followed redirects. That's the thing you actually want to scrape.

Diagnose

How to tell if a site needs rendering

1
Open the page, then view its raw source (Ctrl+U or curl the URL).
2
Search the source for a sentence you can see on the page.
3
If it's there, the site is server-rendered — a plain fetch is enough.
4
If the source is a small shell with a large script bundle and none of the visible text, the content is client-rendered — you need a real browser to get it.

Solve

Getting the rendered DOM

The fix is to load the page in a real, headless browser — the same engine as Chrome — let its JavaScript execute, and read the resulting HTML. That's what a rendered-HTML fetcher does: navigate, wait for the content, follow redirects, and return the DOM a visitor sees rather than the empty shell the server sent.

Use it to scrape single-page apps that ship an empty HTML shell.
Grab the post-JavaScript DOM for parsing, extraction or conversion to Markdown.
Debug the gap between a page's source and what it actually renders.

Timing

The content arrives after the page has "loaded"

Rendering is necessary but not always sufficient. Plenty of single-page apps finish loading — the load event fires, the network goes quiet — and only then fetch their data and draw it. Read the DOM at that moment and you get a skeleton: grey placeholder boxes, a spinner, an empty table.

Waiting a fixed number of seconds is the usual workaround, and the wrong one: too short on a slow day, wasted time on a fast one. The reliable fix is to wait for the thing you want — a CSS selector that only exists once the content is there, such as the article body, the first table row or the price element.

1
Open the page in a browser and inspect the content you need.
2
Pick a selector for an element that appears only when that content has arrived — not the container, which is often rendered empty first.
3
Tell your headless browser to wait for it (page.waitForSelector in Puppeteer, locator.waitFor in Playwright), with a timeout, before reading the DOM.
4
Treat a timeout as a real failure: the content never arrived, and an empty result would be a silent bug.

With URLpipe that wait is one field. page_options.wait_for_selector holds the read until the element exists, for up to 10 seconds; an element that never appears fails the request, and a failed request costs nothing. For content that arrives on a timer rather than into an element, page_options.delay adds up to 10 seconds after the page settles. Both are in the page options docs.

POST /html

{
  "url": "https://example.com/dashboard/report",
  "sync": true,
  "page_options": {
    "wait_for_selector": "table.results tbody tr"
  }
}

When it's still broken

The page renders — but content is missing

Sometimes even the rendered DOM is incomplete: a section never appears, or the data is blank. That almost always means the JavaScript that builds it failed — a script error, a blocked request, a failed API call.

The fastest way to find out is to capture the page's console output while it loads. Errors, warnings and uncaught exceptions point straight at the broken script or request, instead of leaving you guessing why a render came back half-empty. The guide on finding JavaScript errors on any site covers what a capture sees and what it misses.

FAQ

Frequently asked questions

How do I tell if a site is client-rendered?

Compare what you see in the browser with the page's raw source (Ctrl+U) or a curl of the URL. If the source is a small shell with a big <script> bundle and none of the visible text, the content is rendered client-side and you'll need a real browser to scrape it.

Why does curl work on some sites but not others?

Server-rendered and static sites send complete HTML, so curl gets everything. Single-page apps (React, Vue, Svelte, Angular) send a shell and build the page in the browser, so curl gets almost nothing. Many sites are a mix.

What does "the rendered DOM" mean?

The Document Object Model after the browser has parsed the HTML, run the JavaScript, applied changes and followed redirects — i.e. what you'd see in DevTools' Elements panel, not what "view source" shows.

What if the content loads after the page has finished loading?

Wait for the element you need rather than for a fixed time: in Puppeteer or Playwright, wait for its CSS selector before reading the DOM. With URLpipe, page_options.wait_for_selector does the same for up to 10 seconds, and a page where it never appears fails without being billed.

How do console errors help when scraping?

If a rendered page is still missing content, the JavaScript that builds it probably failed. Capturing the console output surfaces the errors and uncaught exceptions behind an incomplete render — often faster than guessing.

Try it yourself

Free tools for this

No signup — run these on a real page right now, then call the same endpoint from your code.

/html API docs /console API docs

Keep reading

Verify webhook signatures (HMAC)Polite scraping: robots.txt and pacing Idempotency keys explained Cookie banners and ads in scraped pages HTML to Markdown for LLMs All guides

Put this into practice.

Each of the eight kinds of data URLpipe returns has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 1,000 credits a month, no card.

Start free Email the team