Proxy Pools for Web Scraping: Gateway vs List, Health, and Capacity

This guide is for engineers operating web-scraping workers that already know how to send traffic through a proxy and now need to decide how routes enter a pool, how they are selected, when they are quarantined, and when a job must stop rather than rotate again.

It deliberately does not give a universal "requests per IP" number. Capacity depends on the target, provider, session model, geography, and the workload you measure. For sizing, use How Many Proxies Do You Need for Web Scraping?. For the wider control-plane design, see Web Scraping Proxy Architecture.

What a proxy pool should own

A useful proxy pool owns more than endpoint strings. At minimum, it should be able to answer:

• which routes are currently eligible for a job
• which capabilities each route provides, such as protocol and geography
• whether a route is already pinned to a sticky workflow
• whether recent failures justify temporary quarantine
• how many attempts and route changes remain for the job
• which failures are configuration errors, target responses, or transport failures
• whether a result is valid for the requested market and task

The pool should not silently turn every failure into another identity change. Rotation is one recovery tool, not an authorization bypass or a substitute for rate control.

Managed gateway vs explicit pool

The two common operating models have different responsibilities.

A gateway can still expose country, city, ASN, or sticky-session parameters, but those are provider-specific capabilities. Do not assume a routing token used by one provider exists on another.

Likewise, an explicit list does not automatically mean every endpoint represents a unique exit identity at every moment. Verify the behavior of the actual product you use.

Keep pool membership separate from job routing

Treat inventory and job assignment as different layers.

Route inventory answers what exists. Eligibility answers what can satisfy this job. Selection chooses among eligible routes. Session binding preserves continuity when the workflow requires it.

That separation prevents a common mistake: choosing a healthy route that cannot satisfy the required country, protocol, or session behavior.

Define an explicit route record

For an application-managed pool, keep capabilities and runtime health together without mixing them into credentials.

{
  "routeId": "route-017",
  "provider": "provider-a",
  "endpointRef": "secret://proxy/provider-a/route-017",
  "protocols": ["http", "https"],
  "country": "US",
  "region": null,
  "city": null,
  "state": "healthy",
  "quarantineUntil": null,
  "activeSessions": 2,
  "recent": {
    "connectFailures": 0,
    "proxyAuthFailures": 0,
    "target429": 1,
    "wrongGeo": 0
  }
}

endpointRef is intentionally a secret reference rather than a URL containing credentials. Logs, traces, bug reports, and analytics should not receive raw proxy passwords or reusable session tokens.

Filter before you rank

Selection should start with hard requirements.

Example eligibility order:

1. route is enabled and not quarantined
2. protocol matches
3. required country matches
4. required region/city matches when the product actually supports it
5. sticky session is either already bound to this route or the route can accept a new binding
6. provider/account limits permit another assignment

Only after those checks should you rank eligible routes using softer signals such as recent transport health or current load.

Never silently downgrade a required city to a country-only route. Return an explicit no_eligible_route result so the caller can stop, relax the requirement deliberately, or ask for human review.

A small Python pool selector

The following example uses only the Python standard library. It demonstrates eligibility, temporary quarantine, least-loaded selection, and sticky binding. The thresholds are example policy values, not universal recommendations.

from __future__ import annotations

from dataclasses import dataclass
from time import monotonic
from typing import Dict, Iterable, Optional


@dataclass
class Route:
    route_id: str
    country: str
    protocol: str = "http"
    active_sessions: int = 0
    enabled: bool = True
    quarantine_until: float = 0.0

    def available(self, *, country: str, protocol: str, now: float) -> bool:
        return (
            self.enabled
            and now >= self.quarantine_until
            and self.country.upper() == country.upper()
            and self.protocol == protocol
        )


class ProxyPool:
    def __init__(self, routes: Iterable[Route]) -> None:
        self.routes: Dict[str, Route] = {route.route_id: route for route in routes}
        self.sessions: Dict[str, str] = {}

    def acquire(
        self,
        *,
        country: str,
        protocol: str = "http",
        session_key: Optional[str] = None,
    ) -> Route:
        now = monotonic()

        if session_key and session_key in self.sessions:
            route_id = self.sessions[session_key]
            route = self.routes.get(route_id)
            if route and route.available(country=country, protocol=protocol, now=now):
                return route
            raise RuntimeError("sticky route is no longer eligible")

        candidates = [
            route
            for route in self.routes.values()
            if route.available(country=country, protocol=protocol, now=now)
        ]
        if not candidates:
            raise RuntimeError("no eligible proxy route")

        route = min(candidates, key=lambda item: (item.active_sessions, item.route_id))

        if session_key:
            self.sessions[session_key] = route.route_id
            route.active_sessions += 1

        return route

    def quarantine(self, route_id: str, seconds: float) -> None:
        if seconds <= 0:
            raise ValueError("quarantine seconds must be positive")
        route = self.routes[route_id]
        route.quarantine_until = monotonic() + seconds

    def release_session(self, session_key: str) -> None:
        route_id = self.sessions.pop(session_key, None)
        if route_id is None:
            return
        route = self.routes[route_id]
        route.active_sessions = max(0, route.active_sessions - 1)

Important boundaries of this example:

• it is an in-memory teaching implementation, not a distributed lock service
• it does not persist session ownership across process restarts
• it does not infer whether a target response is a proxy failure
• the caller must classify outcomes before calling quarantine()
• multi-worker production systems need atomic/shared session ownership or deterministic partitioning

Do not quarantine a route for every 4xx response

A pool needs a failure taxonomy before it can maintain useful health state.

HTTP 407 Proxy Authentication Required is specifically a proxy authentication challenge, and the proxy must send a Proxy-Authenticate challenge. Treating it as a generic bad-exit signal can cause useless rotation across otherwise healthy routes.[1]↗

HTTP 429 Too Many Requests means the client has sent too many requests in a given amount of time; the response may include Retry-After. The standard deliberately does not require rate limiting to be keyed only by IP, so rotating identities is not a standards-based substitute for backing off.[2]↗

Health should be scoped to the failure

Avoid a single global healthy=false bit.

A route can be:

• unreachable from one worker region but reachable from another
• valid for one protocol but misconfigured for another
• geographically correct for one job and wrong for another
• transport-healthy while the target is rate-limiting the workload

Useful health keys can include:

provider + endpoint + protocol + worker_region + target_class + time_window

Do not make the key more granular than your traffic can support statistically. A health metric based on one request is usually an observation, not a trend.

Quarantine needs an exit path

A route that enters quarantine must have a documented way back.

A practical state machine is:

The exact thresholds and quarantine duration should come from your own failure history and provider behavior. Do not publish or copy arbitrary constants as if they are universal safe values.

Sticky sessions consume pool capacity differently

A stateless request can release its route immediately after the request. A sticky workflow may reserve an identity while cookies, server-side session state, or a multi-page task remain active.

Track at least:

• active sticky sessions
• session age
• expiry time
• owning job
• route ID
• release reason

A session that expires or fails must be released. Otherwise an in-memory or distributed session map can grow indefinitely even when the underlying work is finished.

For a quantitative sizing model, keep this page focused on operations and use How Many Proxies Do You Need for Web Scraping? for measured capacity calculations.

Retry budgets belong to the job, not only the route

Without a job-level budget, a pool can create an accidental retry storm by repeatedly choosing another healthy-looking identity.

Example policy:

retry_budget:
  max_attempts: 3          # example value; measure for your workload
  max_route_changes: 2     # example value
  max_elapsed_seconds: 90  # example value
  retryable:
    - proxy_connect_timeout
    - proxy_connection_reset
  stop:
    - proxy_407
    - target_401
    - explicit_policy_denial
    - no_eligible_route

The important part is not these example numbers. The important part is that attempts, route changes, elapsed time, and bytes are finite and observable.

Verify pool behavior with control tests

Before attributing a failure to the pool, run a small diagnostic matrix.

When you control the diagnostic endpoint, return the observed source IP and any trusted geo metadata. Do not rely on a public IP-check service as your sole production health oracle.

Metrics that make a pool debuggable

Record metrics tied to a useful result, not only request completion:

• eligible routes per job class
• route-selection count
• active sticky sessions
• quarantine entries and exits
• proxy connect failure rate
• proxy 407 rate
• target 429 rate
• wrong-geo result rate
• retries per usable result
• route changes per usable result
• bytes per usable result
• p50/p95 time to usable result

Keep target responses separate from proxy transport failures. Otherwise a target-side rate limit can incorrectly poison the health score of the entire pool.

For credential logging rules, see How to Redact Proxy Credentials from Logs, Traces, and Bug Reports.

Pool segmentation: isolate when the policies differ

Separate pools or logical segments when workloads require materially different rules, for example:

• different countries or cities
• stateful browser sessions vs stateless HTTP jobs
• different providers
• production vs staging
• workloads with different authorization or policy boundaries

Do not segment only to create the appearance of more identities. Segmentation should correspond to a routing, security, capacity, or operational reason.

Compliance and stop conditions

Proxy pooling does not change whether a request is authorized or permitted.

Before operating a scraper at scale:

1. confirm you are authorized to collect the target data
2. review the site's terms and applicable contractual restrictions
3. handle personal or sensitive data according to your privacy and retention obligations
4. inspect robots.txt where relevant to crawler behavior
5. stop or escalate on explicit authorization failures instead of automatically rotating
6. rate-limit workloads to avoid causing service disruption

RFC 9309 standardizes the Robots Exclusion Protocol and explicitly states that its rules are not a form of access authorization. Treat robots rules as crawler instructions, while authorization and legal permission remain separate questions.[3]↗

Production checklist

Before relying on a proxy pool:

1. Inventory and credentials are stored separately.
2. Eligibility enforces protocol and requested geography before selection.
3. Sticky session ownership and expiry are explicit.
4. No eligible route fails closed instead of silently downgrading requirements.
5. 407, transport failures, 429, access denials, wrong output, and parser failures are classified separately.
6. Quarantined routes have expiry and controlled recovery behavior.
7. Every job has finite attempt, route-change, time, and traffic budgets.
8. Control tests can separate target, worker, and proxy failures.
9. Credentials and reusable session identifiers are redacted from logs.
10. Metrics report usable output and route changes, not only HTTP success.
11. Authorization, terms, privacy, and rate limits are part of the operating policy.

FAQ

Is a bigger proxy pool always better?

No. A larger inventory does not fix incorrect routing, excessive request rate, broken session handling, parser errors, or authorization problems. Measure whether additional eligible routes improve usable output for the actual workload.

Should I rotate after every failed request?

No. Classify the failure first. A proxy connect failure may justify a different route. A 407 usually requires fixing proxy authentication. A target 429 calls for rate reduction/backoff, and an explicit access denial should trigger a stop/review path rather than unbounded rotation.

Should each worker own its own pool?

Not necessarily. Per-worker pools are simple but can duplicate health state and session ownership. A shared pool can coordinate better but needs atomic state or deterministic partitioning. Choose based on your concurrency model and failure domain.

Can a gateway replace pool management entirely?

It can remove explicit exit inventory management, but your application still owns job policy, session intent, retry limits, result validation, credential safety, and stop conditions.

How do I decide how many routes I need?

Measure the workload rather than using a universal per-IP number. The dedicated sizing guide is How Many Proxies Do You Need for Web Scraping?.

Related BytesFlows resources

• Web Scraping Proxy Architecture
• How Many Proxies Do You Need for Web Scraping?
• How Proxy Rotation Works
• Proxy Rotation Strategies
• Playwright Proxy Errors
• Web Scraping Proxies

Validate the pool against real workload conditions

Treat these patterns as an operating model, then verify provider capabilities, exit behavior, capacity, session limits, and target-specific acceptance with the product and authorized workload you actually run. Pool health should come from observed evidence rather than assumptions copied from another environment.