Web Scraping Proxy Architecture: Session Routing, Retries, and Observability

As a scraping system grows, proxy behavior affects correctness, not only connectivity. A route can return the wrong country, break a sticky workflow, inflate retries, or produce a valid HTTP status with unusable content.

Reference architecture

The scraper asks for a route intent, not an arbitrary IP.

Route intent contract

{
  "jobId": "catalog-us-001",
  "targetClass": "ecommerce-public",
  "country": "US",
  "region": null,
  "city": null,
  "protocol": "http",
  "sessionMode": "sticky",
  "sessionKey": "catalog-us-001",
  "maxSessionMinutes": 10,
  "expectedBytes": 500000,
  "sensitivity": "browser"
}

The policy service validates whether the requested geography and session mode are available and allowed for the workload.

Separate control plane and data plane

Control plane responsibilities:

• policy configuration
• account and credential management
• route class selection
• health aggregation
• session ownership
• quotas and budgets
• incident controls

Data plane responsibilities:

• accept proxy connections
• authenticate
• select or bind an exit route
• relay traffic
• count bytes
• emit connection outcomes

Keep per-request forwarding paths free from slow database lookups when possible. Distribute signed or versioned configuration to gateways and cache decisions with bounded lifetimes.

Session broker

The session broker maps a logical workflow to a route:

{
  "sessionKey": "checkout-qa-002",
  "routeId": "redacted",
  "country": "GB",
  "createdAt": "2026-08-07T00:13:00+08:00",
  "expiresAt": "2026-08-07T00:23:00+08:00",
  "state": "active",
  "ownerJob": "checkout-qa-002"
}

Rules:

• one stateful workflow owns one sticky session
• independent jobs do not inherit cookies or session IDs accidentally
• expiry is explicit
• failed routes can be revoked
• retries cannot create unbounded sticky-session growth

Rotation policy

Rotation should happen at a business boundary:

• per independent URL job
• per search query snapshot
• per catalog page group
• after an unhealthy route classification

Do not rotate in the middle of login, checkout, form, or paginated tasks that require continuity.

Response classification

A classifier should combine transport and application evidence:

{
  "transport": "success",
  "proxyAuth": "accepted",
  "httpStatus": 200,
  "pageClass": "challenge",
  "geoMatch": true,
  "businessOutput": "missing",
  "retryClass": "review_or_backoff"
}

HTTP 200 is not the same as useful output.

Failure taxonomy

Retry budget

Give every job a finite budget:

{
  "maxAttempts": 3,
  "maxRouteChanges": 2,
  "maxElapsedSeconds": 90,
  "maxBytes": 5000000,
  "retryable": ["connect_timeout", "route_reset", "target_503"],
  "nonRetryable": ["proxy_407", "target_401", "policy_denied"]
}

Use exponential backoff with jitter for transient failures. A retry should change a meaningful condition or wait for recovery.

Route health is multi-dimensional

Track health by:

• gateway
• exit route or pool
• country and ASN
• target class
• protocol
• client version
• time window

A route can be healthy for a neutral endpoint and poor for one target class. Avoid a global healthy/unhealthy bit that hides context.

Circuit breakers

Open a breaker when a narrow route segment shows elevated transport failures or wrong-output results. Do not trip an entire country because one target changed its page.

Breaker key example:

provider + country + target_class + protocol

States should be visible: closed, open, half-open. Probe with a small controlled workload before restoring normal volume.

Observability

Connection event:

{
  "connectionId": "conn-001",
  "jobId": "catalog-us-001",
  "routePolicy": "residential-us-sticky",
  "requestedCountry": "US",
  "observedCountry": "US",
  "status": 200,
  "pageClass": "expected",
  "uploadBytes": 2150,
  "downloadBytes": 184220,
  "durationMs": 2810,
  "attempt": 1
}

Redact credentials, full session tokens, and sensitive URLs.

Metrics tied to useful output

• successful business records
• bytes per successful record
• retries per successful record
• wrong-geo rate
• challenge rate
• session break rate
• 407 rate
• connection failure rate
• p50 and p95 time to useful output
• route concentration
• unique exit distribution where relevant

Raw request success can hide expensive bad pages.

Credential architecture

• keep provider credentials in a secret manager
• issue scoped sub-credentials where supported
• rotate credentials in stages
• avoid embedding passwords in source or logs
• use distinct credentials for environments
• audit consumers before revocation
• maintain a rollback window where provider capabilities permit

Multi-provider design

A provider abstraction can improve resilience, but lowest-common-denominator design can hide important differences. Model capabilities explicitly:

{
  "provider": "example",
  "protocols": ["http", "socks5"],
  "geoLevels": ["country", "city"],
  "stickyMaxMinutes": 30,
  "supportsAsn": true,
  "authModes": ["userpass"]
}

Fail closed when a required capability is unavailable. Do not silently downgrade city targeting to country targeting.

Browser workers

For Playwright or Puppeteer:

• one BrowserContext per logical identity when browser-state isolation is required
• proxy session and cookie lifetime aligned intentionally rather than assumed
• bounded pages per browser
• close contexts in finally; Playwright documents contexts as isolated, non-persistent sessions and recommends disposing them when no longer needed[3]↗
• collect trace only for sampled or failed jobs
• classify navigation, HTTP, and business-output failures separately

A proxy changes network routing and the source IP observed by the destination. It does not automatically change cookies, account history, TLS/browser implementation, JavaScript-visible properties, device state, or every fingerprint signal. Treat those as separate identity and application-state dimensions.

Authorization and stop conditions

A production retry engine also needs explicit do-not-retry boundaries. Stop the automated workflow and require review when:

• the target explicitly denies access and the job owner cannot confirm authorization
• continuing would require bypassing an access control, challenge, or security mechanism
• the collected data moves outside the approved purpose, retention, or privacy boundary
• credentials, tokens, or sensitive URLs appear in logs or evidence
• retries are no longer changing a meaningful condition

Proxy rotation is a routing tool, not permission to defeat target controls. Rate limiting is also not necessarily IP-scoped: RFC 6585 notes that a server may identify a rate-limited user through authentication credentials, cookies, or other scope.[2]↗

Cost controls

Estimate:

monthly_cost = valid_jobs × bytes_per_valid_job × price_per_byte × retry_multiplier

Measure rather than guessing. Browser assets, screenshots, retries, and wrong pages can dominate usage. Block unnecessary resources only when doing so does not change required output.

Deployment checklist

1. Route intent schema defined.
2. Sticky session ownership enforced.
3. Retry and byte budgets configured.
4. Response classifier tested with challenge and wrong-geo pages.
5. Credentials redacted.
6. Health dimensions include target class.
7. Circuit breakers scoped narrowly.
8. Direct and proxy control tests available.
9. Metrics report useful output.
10. Human review exists for policy and data-quality failures.

Related BytesFlows resources

• Web scraping architecture design
• Proxy rotation strategy
• Proxy pools for web scraping
• Python proxy scraping
• Web scraping proxies

Practical limits

Treat the architecture as implementation guidance rather than a benchmark. Measure capacity, protocol behavior, exit assignment, session duration, geography accuracy, target compatibility, and credential handling against the provider and target workflow you actually use.

RFC 9110 defines the proxy-authentication behavior referenced above, RFC 6585 covers HTTP 429, and Playwright's BrowserContext documentation describes the browser lifecycle used in the examples. The observability fields shown here are an application schema; map them to current OpenTelemetry HTTP semantic conventions when exporting generic telemetry instead of assuming the example JSON is portable as-is.[4]↗