How Many Proxies Do You Need for Web Scraping? A Measurement-Based Sizing Guide

First Decide What “Proxy Count” Means

The phrase how many proxies do I need? can describe two different systems:

• Managed rotating gateway: your crawler connects to one endpoint while the provider selects underlying exits. You usually size concurrency, sticky sessions, traffic volume, and provider limits—not a literal list of IPs.
• Explicit proxy or static-IP list: your system chooses individual identities. Here, an estimated identity count can be useful, but the safe workload per identity must come from measurement against your authorized target.

This distinction matters. A gateway backed by a large pool is not equivalent to one IP, and a list of 100 endpoints is not automatically 100 healthy, interchangeable identities.

For pool architecture and gateway-vs-list tradeoffs, see Proxy Pools for Web Scraping. For a broader routing design, see Web Scraping Proxy Architecture.

The Sizing Model

For an explicit identity pool, start with a measured capacity model:

Then account for sticky workflows:

These formulas are only useful when the denominator is measured from your own workload. Do not substitute a generic “requests per IP” number from a blog post.

If your provider uses a managed rotating gateway, replace identity_slots with the provider capacity dimensions you can actually control: maximum concurrency, sticky-session availability, geography, bandwidth, request quotas, or connection limits.

What to Measure in a Pilot

A useful pilot records both network outcomes and business outcomes.

HTTP 429 Too Many Requests is defined for rate limiting and may include a Retry-After header. Treat that as a backoff signal rather than evidence that you simply need more IPs. See RFC 6585↗.

A Reproducible Pilot Procedure

Use a small representative sample before sizing production capacity.

1. Define the authorized target set. Record domains, markets, page classes, and whether the workflow is HTTP-only or browser-based.
2. Define success. A successful request should produce the expected page or business record, not merely HTTP 200.
3. Start at low load. Use one route or a deliberately small route set so you can observe failure behavior clearly.
4. Increase load gradually. Change one variable at a time: request rate, concurrency, session duration, or geography.
5. Stop on policy or safety signals. Do not turn 401/403, explicit access denial, account warnings, or sustained 429 responses into an IP-rotation loop.
6. Choose an operating point below the failure region. Record the exact conditions: target class, client, request mix, geography, session mode, and time window.
7. Repeat across representative workloads. Product pages, search pages, browser flows, and regional routes can have different capacity profiles.
8. Re-run after material changes. Parser changes, browser upgrades, new geographies, provider changes, or target changes can invalidate the old measurement.

For crawler operators, also review the target’s terms and applicable policies. The standardized Robots Exclusion Protocol is specified in RFC 9309↗; robots rules are crawler instructions, not access authorization.

Example: Explicit Proxy List

Assume your own pilot produced these illustrative inputs:

• target workload: 12,000 requests/hour
• measured operating point: 240 requests per identity per hour
• concurrent sticky sessions: 18
• additional capacity margin chosen by your team: 20%

The measured baseline is:

Applying the illustrative 20% margin gives 60 identity slots. Since 60 is also greater than 18 sticky sessions, the working capacity estimate is 60.

The numbers above are examples, not BytesFlows benchmarks and not recommended universal limits.

Calculator Script

This script deliberately requires measured inputs. It does not contain target-specific rate assumptions.

from __future__ import annotations

import math


def estimate_identity_slots(
    target_requests_per_window: int,
    measured_requests_per_identity_per_window: float,
    concurrent_sticky_sessions: int = 0,
    capacity_margin_ratio: float = 0.0,
) -> int:
    if target_requests_per_window <= 0:
        raise ValueError("target_requests_per_window must be > 0")
    if measured_requests_per_identity_per_window <= 0:
        raise ValueError("measured_requests_per_identity_per_window must be > 0")
    if concurrent_sticky_sessions < 0:
        raise ValueError("concurrent_sticky_sessions must be >= 0")
    if capacity_margin_ratio < 0:
        raise ValueError("capacity_margin_ratio must be >= 0")

    throughput_slots = math.ceil(
        target_requests_per_window / measured_requests_per_identity_per_window
    )
    with_margin = math.ceil(throughput_slots * (1 + capacity_margin_ratio))

    return max(with_margin, concurrent_sticky_sessions)


if __name__ == "__main__":
    # Illustrative inputs only. Replace them with measurements from your pilot.
    slots = estimate_identity_slots(
        target_requests_per_window=12_000,
        measured_requests_per_identity_per_window=240,
        concurrent_sticky_sessions=18,
        capacity_margin_ratio=0.20,
    )
    print({"estimated_identity_slots": slots})

Expected output for the illustrative inputs:

{"estimated_identity_slots": 60}

Rotating Gateway: What to Size Instead

With a managed rotating residential gateway, asking for a literal IP count can be misleading. Track the capacity controls the provider exposes and the outcomes you observe.

For bandwidth planning, use How Much Proxy Bandwidth Do You Need for Web Scraping?.

Sticky Sessions Need Their Own Capacity Check

A throughput-only formula can under-size stateful workloads.

If your architecture assigns one proxy identity to each logical sticky workflow, your minimum identity capacity cannot be lower than peak concurrent sticky sessions. Examples include:

• multi-page browser journeys
• cart or delivery-region checks
• authenticated QA where authorization permits automation
• paginated tasks whose server-side state depends on continuity

Do not rotate midway through a workflow merely because a timer expired. Define the session boundary around the task and release the identity when the task finishes or a bounded failure policy stops it.

Failure Classification Before Adding Capacity

More proxy capacity cannot fix every failure.

Validation Checklist

Before buying or reserving more proxy capacity, confirm:

FAQ

Is there a safe universal number of requests per IP?

No. A defensible value must come from your authorized target, workload, geography, client behavior, and operating conditions. Fixed cross-site ranges are too easy to misuse and too difficult to verify.

Do I need one proxy per worker?

Not necessarily. Workers are compute units; proxy identities are network-routing capacity. One worker may use many rotating exits, while several low-rate workers may share a gateway. Size from measured workload and session requirements.

Does more IPs always reduce blocks?

No. Failures can come from target policy, authentication, request behavior, browser state, parser bugs, network faults, or rate limits. Adding identities without classifying the failure can increase cost and retry noise.

How many proxies do I need for Playwright?

Start with peak concurrent logical browser sessions and the provider’s supported session model, then validate with a representative browser pilot. Browser resource loading also affects bandwidth, so capacity and GB planning should be measured together.

Should I rotate after every failed request?

No. Retry only when the failure class is actually retryable. Authentication errors, explicit access denials, policy failures, and repeated rate limits should not become automatic rotation loops.

Related BytesFlows Guides

• Proxy Pools for Web Scraping
• Web Scraping Proxy Architecture
• Proxy Rotation Strategy
• How Much Proxy Bandwidth Do You Need for Web Scraping?
• Web Scraping Proxies

Validate the sizing model before scaling

Use this method with measurements from your own authorized workload and current provider constraints. Safe request rate, usable-result rate, concurrency, sticky-session demand, and required capacity vary by target and operating conditions, so re-run the pilot whenever those conditions materially change.