Guide

Cookie banners and ads getting the page a reader sees

A fresh headless browser is a first-time visitor from nowhere in particular, so it gets every consent wall, ad and newsletter pop-up a site has. Here is how to get the page underneath.

By Roger Campos · Last updated: September 2026

TL;DR

Scraped pages fill with cookie banners and ads because an automated browser is always a first-time visitor with no consent stored. Block ad requests before the page loads, remove consent managers' banners and their policy text from the DOM without clicking anything, remove your own list of selectors, and restore scrolling the banner locked. Removing beats hiding: hidden text still leaks.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

Why it happens

An automated browser is the worst-case visitor

Your screenshot is half cookie banner. Your Markdown starts with four paragraphs of consent policy. Your summary mentions "our 847 partners". None of that is a bug in your tool.

A headless browser starts every session empty: no cookies, no stored consent, no history. To a site, that is a first-time visitor, every single time — so it gets the full treatment. The consent banner, because no choice has been stored. The newsletter pop-up, because it hasn't been dismissed. Every ad slot, because there's nothing to cap them.

Where you fetch from matters too. A visit from a European address — which is where URLpipe fetches from, and where any fetcher keeping data in the EU is likely to — gets the GDPR consent flow on most commercial sites, often as a wall that covers the page entirely until someone chooses.

The damage

What banners and ads do to each kind of output

Output	What goes wrong
Screenshot	The banner covers the page; a consent wall covers all of it. Many banners also lock scrolling, so a full-page capture is one viewport tall.
Markdown / text	Consent-policy text, vendor lists and ad labels become part of the content — often at the top, where it's weighted most.
Summary / keywords	The model summarizes the cookie policy, or returns "privacy" and "partners" as the page's keywords.
Console capture	Ad and tracking scripts contribute most of the errors, burying the page's own.
Load time	Ad auctions and trackers pull in more scripts. Pages take longer, and time out more often.

The options

Five ways to deal with it, and which to use

Approach	How	Trade-off
Click "Accept"	Find the button and click it	Gives consent on someone's behalf, and loads the trackers the banner was holding back. Different on every site.
Pre-set a consent cookie	Set the consent manager's cookie before loading	Per-vendor and per-site formats that change; still records a choice nobody made.
Hide with CSS	display: none on the banner	Fine for a screenshot. The text is still in the DOM, so text extraction still reads it.
Remove the elements	Delete the banner's nodes from the DOM before reading	Needs selectors for each consent manager — but gone is gone, from every output.
Block the requests	Refuse ad and consent scripts at the network level	Fastest pages; a site that depends on those scripts may render differently.

The combination that works best is the last two: block ad requests before the page loads, and remove consent banners from the DOM once it has. Nothing is clicked, so no consent is given, and nothing is merely hidden, so nothing leaks into the text. A handful of consent managers — OneTrust, Cookiebot, Usercentrics, Didomi, Quantcast, Sourcepoint and a few more — serve most of the web's banners, so a short list of their container selectors covers a large share of sites.

Doing it yourself

Blocking and removing in Playwright

Block ad requests, remove banners, restore scrolling

import { chromium } from "playwright";

const AD_HOSTS = ["doubleclick.net", "googlesyndication.com", "adnxs.com", "criteo.com"];
const BANNERS = ["#onetrust-consent-sdk", "#CybotCookiebotDialog", "#usercentrics-root",
                 "#didomi-host", ".qc-cmp2-container", "[id^='sp_message_container']"];

const browser = await chromium.launch();
const page = await browser.newPage();

// 1. Refuse ad requests before they start.
await page.route("**/*", (route) => {
  const host = new URL(route.request().url()).hostname;
  return AD_HOSTS.some((ad) => host === ad || host.endsWith("." + ad)) ? route.abort() : route.continue();
});

await page.goto("https://example.com/article", { waitUntil: "load" });

// 2. Remove consent banners, and give the page its scrolling back.
await page.evaluate((selectors) => {
  document.querySelectorAll(selectors.join(",")).forEach((el) => el.remove());
  for (const el of [document.documentElement, document.body]) {
    el.style.setProperty("overflow", "auto", "important");
    el.style.setProperty("position", "static", "important");
  }
}, BANNERS);

await page.screenshot({ path: "article.png", fullPage: true });
const html = await page.content();
await browser.close();

The lists are the part that needs upkeep. Ad hosts are better taken from a maintained filter list than typed by hand, and consent managers rename their containers now and then. Some banners load late, after the first removal pass — if one reappears, remove again after a short wait, or watch for it with a MutationObserver.

The subtle part

Why removing beats hiding

It's tempting to inject display: none and move on — the screenshot looks right. But most HTML-to-text and HTML-to-Markdown converters don't read CSS. They walk the DOM, and a hidden banner is still in the DOM. Its text ends up in your output, and a consent manager's preference panel can be thousands of words of policy.

On onetrust.com, removing the consent manager's elements takes about 16,000 characters of cookie policy out of the page's text. That is text a model would otherwise read as the page.

What this won't get past

Paywalls, login walls and "subscribe to continue" gates aren't banners — the content isn't in the page to uncover. Removing the overlay just shows you the teaser underneath.

With URLpipe

page_options: the page a reader sees, in one field

URLpipe does the block-and-remove approach for you through page_options, on every endpoint that loads the page: block_ads refuses the major ad networks' requests before the page loads and removes the slots they'd have filled; block_cookie_banners removes the banners of the consent managers that serve most of the web, their policy text included, without clicking anything; remove_selectors takes up to 50 of your own — a chat widget, a promo bar, a related-posts list. For screenshots, the scroll lock a removed banner leaves behind is lifted too, so a full-page capture isn't cut off at the fold.

What's removed is gone from every result — the HTML, the Markdown, the summary, the keywords, the console and the screenshot. On one news site, block_ads roughly halved the time the page took to load. The options are included in each operation's usual credits, and a page with its ads removed is cached as its own result. To change only how a screenshot looks, while keeping the element in the HTML, use screenshot_options.hide_selectors instead. The page options docs have every field.

POST /screenshot

{
  "url": "https://example.com/article",
  "sync": true,
  "page_options": {
    "block_ads": true,
    "block_cookie_banners": true,
    "remove_selectors": [".newsletter-modal", "#chat-widget"]
  }
}

FAQ

Frequently asked questions

Why do my screenshots show a cookie banner?

Each headless browser session starts with no cookies, so every site treats it as a first visit and shows its consent banner. Remove the banner's elements before capturing, or it covers the page.

Should my scraper click "Accept" on cookie banners?

Clicking gives consent — on someone's behalf, for tracking you have no use for — and it loads the very scripts the banner was holding back. Removing the banner's elements gets you the page without agreeing to anything.

Is hiding a banner with CSS enough?

For a screenshot, often. For text, no: display: none leaves the element in the DOM, so an HTML-to-text or Markdown conversion that doesn't read CSS still picks up the banner's text. Remove the elements instead.

Does blocking ads make pages load faster?

Usually, and sometimes a lot — ad scripts pull in more scripts, auctions and tracking pixels. On one news site we measured, blocking ads roughly halved the load time.

Why can't I scroll the page after removing the banner?

Consent managers often lock scrolling with overflow: hidden on the html or body element while the banner is open. Removing the banner doesn't undo that; set overflow back to auto as well.

Try it yourself

Free tools for this

No signup — run these on a real page right now, then call the same endpoint from your code.

/page-options API docs /screenshot API docs

Keep reading

Rendered vs. raw HTML Verify webhook signatures (HMAC)Polite scraping: robots.txt and pacing Idempotency keys explained HTML to Markdown for LLMs All guides

Put this into practice.

Each of the eight kinds of data URLpipe returns has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 1,000 credits a month, no card.

Start free Email the team