Authenticated scraping guide
Collect from logged-in sites through persistent profiles.
Login-walled data needs a session, and sessions need continuity. Persistent profiles hold the login while your collector works at a pace a real user could sustain.
Direct answer
Log into the target once in an AliasMode profile, then start and connect over CDP for each collection run via the Local API, keeping one account per site per profile and gentle rate limits.
What you will accomplish
Before you start
- Accounts on the target sites that your terms of service permit you to use this way
- AliasMode installed and the Local API available at 127.0.0.1:50400
- A Playwright or Puppeteer collector
- A residential proxy per profile matching the account's usual region
Step-by-step
Confirm you may collect, then how much
Check the site's terms, robots directives, and your account agreement. Decide a sustainable rate — pages per minute and runs per day — before writing code.
One profile per site account
Create an AliasMode profile per target account: scrape·site·account-name. If a site bans an account, the blast radius is one profile.
Match the proxy to the account
Attach a residential proxy in the region the account normally logs in from, and preflight. A familiar IP is the cheapest anti-challenge measure there is.
Seed the login by hand
Log in manually once inside the profile and solve any challenge personally. Collectors never handle the login form; they inherit the session.
Start the session per run
Start the profile through the Local API at the beginning of each run.
const res = await fetch('http://127.0.0.1:50400/api/v1/browser/start?user_id=' + profileId); const { data } = await res.json(); const browser = await chromium.connectOverCDP(data.ws.puppeteer);Collect at a human pace
Throttle requests, randomize pacing modestly, and stop on rate-limit signals rather than pushing through. Long, slow, boring runs beat fast, hot ones.
const context = browser.contexts()[0]; for (const url of queue) { await page.goto(url); // extract, then wait — keep it slow and steady await page.waitForTimeout(3000 + Math.random() * 4000); }Stop and sync
Close the browser and stop the profile after each run so the session is saved cleanly for tomorrow.
await browser.close(); await fetch('http://127.0.0.1:50400/api/v1/browser/stop?user_id=' + profileId);Monitor session health
If a run lands on a login page or challenge, stop that profile, re-seed the login manually, and investigate the rate before resuming.
The AliasMode workflow
Profiles as site credentials
One profile per site account keeps logins, cookies, and fingerprint history attributable to one identity.
Local API as the run switch
Start and stop sessions from your collector's scheduler; nothing runs outside a managed session.
Preflight per run
Confirm the exit IP region before each run so the account never sees a surprise geo.
Verify it worked
- Runs land on logged-in pages with no login flow executed by code.
- Collection rates match the documented throttle, sampled from run logs.
- Stopping a run mid-way leaves a stoppable, resumable state — no orphan browser.
- Account health is stable: no new challenges or warnings across a month of runs.
Cautions
- Scraping can breach a site's terms even when the data itself is public — get the legal basis right for your jurisdiction and data types.
- Never collect personal data without a lawful basis; authenticated access is not consent.
- Pushing through rate limits risks the account and, on shared IPs, the proxy's reputation.
The collection run loop
Pre-run
Preflight proxy, confirm rate limits, check yesterday's run ended clean.
Start
Local API start returns the CDP endpoint; connect your collector.
Collect
Throttled extraction with stop-on-challenge guards.
Post-run
Stop the profile, store the data, log session health.
Collection guardrails
| Guardrail | Setting | Why |
|---|---|---|
| Request pacing | Seconds between pages, randomized | Machine-gun pacing is the top account killer |
| Daily volume | Cap pages per account per day | Volume spikes read as abuse |
| Stop conditions | Halt on login pages and challenges | Pushing through turns a warning into a ban |
| Run windows | Collect at plausible hours | 4 a.m. marathons are a pattern |
Authenticated scraping FAQ
Is authenticated scraping legal?
It depends on terms, data types, and jurisdiction. Get a legal review for your targets; the tooling here assumes you have the right to collect what you collect.
Why not log in inside the script?
Login forms carry the highest challenge risk. Log in once by hand in the profile, and your code only reuses a stable session.
How many accounts per site should I run?
Prefer one healthy account per site per project. Extra accounts multiply your maintenance and risk for little throughput.
Sources and verification
- IETF · Robots Exclusion Protocol (RFC 9309) (checked September 2026)
- Playwright · Browser.connectOverCDP API reference (checked September 2026)
- Chrome DevTools Protocol · DevTools Protocol documentation (checked September 2026)
Public product details can change after the check date. Facts are re-checked on a monthly cycle.