Page options

Wait for the page you actually want, scroll it to the end of a listing that loads as you go, and take out what you don't want — ads, cookie banners, a newsletter pop-up — before anything is read off it. It all applies to every result: the HTML, the Markdown, the metadata, the summary, the keywords, the console, the screenshot and the PDF.

Send them as one page_options object on any endpoint that loads the page — every one but /lighthouse. They are included in the operation's usual credits.

POST /markdown

{
  "url": "https://example.com/blog/launch",
  "page_options": {
    "block_ads": true,
    "block_cookie_banners": true,
    "block_popups": true,
    "remove_selectors": ["#related"]
  }
}

Options

Name
wait_for_selector
Type
string
Description
Wait for an element matching this CSS selector before reading the page, up to 10 seconds — for a single-page app whose content arrives after the page has loaded. An element that never appears fails the request, and costs nothing.
Name
scroll_to_bottom
Type
boolean
Description
Scroll down the page before reading it, and keep going for as long as scrolling loads more — for a product listing, a search results page or a feed that shows its first items and fetches the rest as a visitor scrolls. See Pages that load as they scroll.
Name
delay
Type
integer
Description
Milliseconds to wait after the page has settled, 0–10000. For content, or console errors, that arrive on a timer.
Name
block_ads
Type
boolean
Description
Block the major ad networks' requests before the page loads, and remove the slots they would have filled. Pages with ads usually load much faster with it.
Name
block_cookie_banners
Type
boolean
Description
Remove the cookie-consent banners of the platforms that serve most of the web's — OneTrust, Cookiebot, Usercentrics, Didomi, Quantcast, Sourcepoint and more — including the cookie policy text their preference panels put in the page. Nothing is clicked, so no consent is given on your behalf.
Name
block_popups
Type
boolean
Description
Remove what a site lays over the page to get a visitor to do something else — a newsletter sign-up, an app-install banner, a discount offer, a survey. The site's own header and menus stay, and so does a sign-in or subscription wall: that is the site deciding who reads the page, and it is still there for you to see.
Name
remove_selectors
Type
array
Description
CSS selectors of your own elements to remove — a chat widget, a promo bar, a related-posts list. Up to 50.

Pages that load as they scroll

Many listings hold only their first screenful when they open and fetch the next batch each time a visitor nears the end. With scroll_to_bottom the page is scrolled a screen at a time to its end, again each time that makes it longer, and then returned to the top — so the HTML, the Markdown and everything else hold the whole list. On a supermarket category page that opens with 20 products, it returns all 250.

POST /markdown

{
  "url": "https://shop.example.com/category/yogurts",
  "page_options": { "scroll_to_bottom": true }
}

Scrolling takes as long as the page takes to load its content, up to 25 seconds; a feed with no end is read as far as it got by then. Only scrolling is done: a Load more button is not pressed, and a list that scrolls inside its own box rather than with the page stays as it opened. With wait_for_selector the element is waited for first, and delay comes after the scroll. A full-page screenshot of a scrolled page is still at most 16,384 px tall.

Removed, not hidden

Everything these options take out is removed from the page, not hidden in it, so it is absent from the HTML and from everything derived from it. On onetrust.com, block_cookie_banners takes about 16,000 characters of cookie policy out of the page's text; on elmundo.es, block_ads roughly halves the time the page takes to load.

To change only how a screenshot looks — hide an element in the image but keep it in the HTML — use screenshot_options.hide_selectors instead.

Stored results

Page options are part of what identifies a result: a page with its ads removed is a different result from the page with them, and each is stored and reused on its own. An option set to its default is the same request as one that leaves it out.

Why not /lighthouse

A Lighthouse audit loads the page for itself, and measures that load. Waiting on it, blocking its requests or trimming it would change the very scores the audit reports, so /lighthouse refuses page_options with invalid_options. In a /scrape, they apply to every operation except the audit.