Guide
Scraping pages behind a login: reuse the session, skip the login
A scraper starts every run as a stranger, so it gets the sign-in page. Sign in once, save what the browser kept, and every later visit starts already signed in.
By Roger Campos · Last updated: October 2026
TL;DR
To scrape a page behind a login, sign in once in a real browser, save the session it holds — its cookies and localStorage, which Playwright writes as a storageState file — and start each scraping run from that file. The site sees a signed-in visitor, nothing logs in again, and multi-factor prompts don't come back until the session ends. Keep the state renewed, keep it on one network, and guard it like a password.
Free plan, no credit card. 1,000 credits a month.
The problem
Why your scraper gets the sign-in page
Being signed in is something your browser remembers, not something the page knows. A scraper starts every run with nothing remembered, so the site treats it as a stranger.
When you sign in, the site hands your browser a token to show on every later request. Most sites put it in a session cookie, usually HttpOnly so the page's own scripts can't read it. Single-page apps often keep an access token in localStorage as well, which their JavaScript attaches to each API call. Together, those are the session.
A headless browser or an HTTP client starts with neither. The site sees an anonymous visitor and does what it does for one: redirects to /login, renders a sign-in form in place of the page, or serves the public version with the account parts missing. None of those is an error, so a scraper that only checks for a 200 will happily store the sign-in page as your data.
The options
Three ways to be signed in
| Approach | How it works | Where it breaks |
|---|---|---|
| Log in on every run | A script fills in the email and password, submits, and scrapes. | Every run is a fresh sign-in: slower, a password stored next to the script, CAPTCHAs, two-factor prompts, "new device" emails, and accounts locked for too many logins. |
| Send a Cookie header | Copy the session cookie into the request of a plain HTTP client. | Pages that render with JavaScript, and apps whose token lives in localStorage, still come back signed out. Renewed cookies are lost unless you parse Set-Cookie yourself. |
| Reuse a saved browser session | Sign in once in a real browser, save its cookies and localStorage, and start every run from them. | Only when the session ends — and then you sign in again, once. |
Reusing the session is the one that holds up. It is also the approach Playwright's own authentication guide recommends for tests: sign in once, then load the saved state into each new browser context.
Step by step
Save a session with Playwright, then reuse it
Playwright calls the saved session a storage state: a JSON file with a cookies list and an origins list holding each origin's localStorage. The quickest way to make one is to sign in by hand in a browser Playwright opens, and let it save what the browser kept when you close the window:
npx playwright codegen --save-storage=state.json https://app.example.com/loginTo do the same from a script — when the sign-in needs a step you want to do yourself, like a two-factor code — open a visible browser, wait while you sign in, and save:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
context = browser.new_context()
context.new_page().goto("https://app.example.com/login")
input("Sign in in the browser window, then press Enter here… ")
context.storage_state(path="state.json")
browser.close()Every scraping run then starts a context from that file, already signed in:
import { chromium } from "playwright";
const browser = await chromium.launch();
const context = await browser.newContext({ storageState: "state.json" });
const page = await context.newPage();
await page.goto("https://app.example.com/account/orders");
console.log(await page.content());
// Keep what the site renewed during the visit for the next run.
await context.storageState({ path: "state.json" });
await browser.close();A state file is not tied to Playwright. It is plain JSON, so cookies exported from your everyday browser can be written in the same shape, and other tools — including URLpipe, below — read it as it is.
Keeping it working
Keeping the session alive
- Write back what the site renews. Many sites issue a fresh session cookie as the old one is used, and expire the old one soon after. Save the state at the end of every run, as above, and a rolling session lasts as long as you keep using it.
- Detect the sign-in page. Check where the visit ended up and what it shows: a redirect to a login URL, or a password field where the content should be, means the session is over. Stop and ask for a new sign-in rather than saving that page as data.
- Stay on one network. Sites watch where a session is used from. One that jumps between a data center and a home connection, or between countries, can look stolen and get ended. Use the session from the same exit every time.
- Don't sign in in parallel. Share one saved session across concurrent visits instead of logging in once per worker; some sites cap active sessions per account and quietly end the oldest.
- Mind sessionStorage. It is not part of a storage state. The rare app that keeps its token there needs the value restored by an init script before its own code runs.
Safety
Treat the state file like a password
Whoever holds the file is signed in as you, until the session ends — no password or second factor needed. Handle it accordingly:
- Keep it out of version control (
.gitignoreit) and out of logs, and encrypt it where it is stored. - Use it only on the site it belongs to. A browser export often also holds cookies for analytics, ad and sign-in-provider domains; don't send those anywhere they weren't set.
- Prefer an account made for the job, with only the access the scraper needs, over a personal or admin account.
- Scrape only accounts you are entitled to use, within the site's terms. Signing in usually means you accepted them.
With URLpipe
Auth profiles: the saved session, kept for you
URLpipe reads the same state file. Upload it once as an auth profile, with the domains it is for, and send auth_profile with any request but /lighthouse — Markdown, HTML, a screenshot, a PDF, metadata, an extraction or a /scrape of several:
jq '{name: "Shop admin", domains: ["shop.example.com"], storage_state: .}' state.json |
curl -X POST https://urlpipe.dev/auth_profiles \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d @-curl -X POST https://urlpipe.dev/markdown \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://shop.example.com/account/orders",
"auth_profile": "ap_7Qm2sX9kLw4RtB1nYc8vDf3H", "sync": true}'Everything this guide describes is done for you. What the site renews during a visit is written back to the profile. Every visit goes out from the one exit the session is used from. A sign-in form in place of the page fails the request without charge and sets the profile's signed_out_at, so you know to sign in again. The state is encrypted at rest, never returned by the API, and used only on the profile's domains; pages fetched with it are stored apart and never served to a request made without it. It costs the same credits as the request without it. The auth profiles docs have the details.
FAQ
Frequently asked questions
It depends on the site's terms, the data and the law where you operate, and signing in usually means you accepted those terms. Use accounts you are entitled to use — your own, or a client's with their permission — and read what the site allows. This is not legal advice.
As long as the site keeps it. Some sessions end when the browser closes or after a fixed time; most are renewed each time they are used. Save the state again after each run and a rolling session stays alive; when it ends, you are shown the sign-in page and need to sign in again.
No. It holds cookies and localStorage. A site that keeps its token in sessionStorage needs it restored with an init script, or a fresh sign-in each run.
A saved session skips it: the second factor was given when you signed in, and the session carries that. You are asked again only when the session ends or the site wants to re-check, which is why reusing a session beats scripting the login.
Yes. DevTools' Application panel lists them, HttpOnly ones included, and cookie-export extensions write them as JSON. Put them in the storageState shape — name, value, domain, path, expires, httpOnly, secure, sameSite — and any tool that reads it can use them.
Try it yourself
Free tools for this
No signup — run these on a real page right now, then call the same endpoint from your code.
Put this into practice.
Each of the eleven kinds of data URLpipe returns has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 1,000 credits a month, no card.