How Much Proxy Bandwidth Do You Need for Web Scraping?

How Much Proxy Bandwidth Do You Need for Web Scraping?

If you need to estimate proxy traffic before a scraping job goes into production, start with measured bytes from a representative sample—not a generic GB-per-page assumption.

A useful forecast has five inputs:

estimated bytes =
  measured bytes per usable result
  × target count
  × market count
  × runs per period
  × retry / failure overhead

The important phrase is usable result. A request that returns 200 OK but contains the wrong market, a challenge page, an empty product page, or a parser failure still consumed bandwidth without producing business value.

This guide shows how to measure HTTP and browser traffic, turn the sample into a monthly GB estimate, and validate the estimate against provider billing before you scale.

For cost-per-result modeling after you have measured traffic, use the Residential Proxy Cost Calculator. That article focuses on cost; this one focuses on capacity planning.

1. Define What Counts as One Result

Do not begin with page count. Define the unit your business actually needs.

Examples:

• one parsed product record
• one SERP result set for a specific country and device profile
• one completed browser workflow
• one price-and-stock observation
• one verified screenshot
• one API-derived record discovered through a browser page

Then define a pass condition. For a product monitor, a usable result might require the expected SKU, currency, market, stock state, and parser version. For a browser workflow, it may require the final state—not just a successful navigation.

This distinction prevents a common budgeting error: dividing traffic by requests instead of by successful outputs.

2. Measure HTTP Traffic First

For plain HTTP collection, measure actual transferred payload on a representative sample.

A simple curl check:

curl \
  --silent \
  --show-error \
  --location \
  --output /dev/null \
  --write-out 'status=%{http_code} downloaded=%{size_download}B uploaded=%{size_upload}B time=%{time_total}s\n' \
  'https://example.com/'

size_download is useful for a quick sample, but it is not a provider billing meter. curl documents it as downloaded payload/body bytes; response headers and other protocol overhead are not included. Provider billing can use a different accounting boundary.

Do not estimate from Content-Length alone. A response can be chunked or otherwise framed without a usable Content-Length, and HTTP transfer framing is not the same thing as your proxy provider's billable-byte definition.

Sample several page classes

Measure page classes separately rather than averaging unrelated targets together:

Collect enough samples to capture normal variation. Do not publish a universal page-size number from one site and reuse it everywhere.

3. Measure Browser Workflows Differently

A browser navigation can trigger document, script, stylesheet, image, font, XHR, fetch, and other requests. Counting only the main HTML response understates browser traffic.

Playwright exposes per-request size information through request.sizes(). The following example records completed network requests and totals their HTTP request/response header and body sizes:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const page = await context.newPage();

let measuredBytes = 0;

page.on('requestfinished', async (request) => {
  try {
    const sizes = await request.sizes();
    measuredBytes +=
      sizes.requestBodySize +
      sizes.requestHeadersSize +
      sizes.responseBodySize +
      sizes.responseHeadersSize;
  } catch (error) {
    console.error('Could not read request sizes:', request.url(), error);
  }
});

try {
  await page.goto('https://example.com/', {
    waitUntil: 'domcontentloaded',
    timeout: 30_000,
  });

  // Wait only for the application-specific state you actually need.
  console.log({
    url: page.url(),
    measuredBytes,
  });
} finally {
  await context.close();
  await browser.close();
}

This is an application-level measurement, not a claim that the number equals billable proxy bytes. TLS, HTTP/2 or HTTP/3 framing, CONNECT traffic, retries below your instrumentation layer, compression behavior, and provider accounting can make the billed total differ.

Resource blocking can reduce traffic—but validate the result

Playwright can abort requests by resource type. For example:

await page.route('**/*', async (route) => {
  const type = route.request().resourceType();

  if (type === 'image' || type === 'media' || type === 'font') {
    await route.abort();
    return;
  }

  await route.continue();
});

Do not assume this is safe for every target. Images, fonts, scripts, or background requests can affect layout, lazy loading, anti-abuse checks, or the specific evidence you need. Compare the blocked and unblocked result and keep the optimization only if the business output remains valid.

4. Convert the Sample into a Forecast

Once you have measured traffic per usable result, forecast by workflow.

Example values below are illustrative inputs, not BytesFlows benchmarks:

measured bytes per usable product record = 1.8 MB
targets per run                         = 12,000
markets                                 = 3
runs per month                           = 20
observed attempt multiplier              = 1.12

Then:

1.8 MB × 12,000 × 3 × 20 × 1.12
= 1,451,520 MB
≈ 1,451.52 GB using decimal GB

If your billing dashboard uses GiB or another unit, convert using the provider's definition instead of assuming that 1 GB = 1 GiB.

The attempt multiplier should come from a pilot run. If 100 usable results require 112 attempts, the observed multiplier is 1.12. Avoid inventing a fixed retry percentage before testing the target.

5. Separate Traffic by Collection Mode

One blended average hides the reason your bill changes.

Track at least these buckets:

A workflow that moves from HTML extraction to full browser rendering can change its traffic profile dramatically without changing the number of final records.

6. Do Not Treat Every Failure as Retryable

Blind retries turn operational problems into bandwidth problems.

For scraping and automation, only collect content you are authorized to access. Respect applicable terms, privacy requirements, rate limits, and explicit stop conditions. Proxies do not remove those obligations.

7. Reconcile Your Measurement with Provider Billing

Before committing to a larger plan, run a controlled pilot and compare two numbers:

1. your application-side measured bytes
2. the provider dashboard's billed traffic for the same window

Record:

• UTC start and end time
• target set
• number of attempts
• number of usable outputs
• application-measured bytes
• provider-reported bytes
• proxy mode and market
• software version

If the numbers differ, do not force them to match by changing the spreadsheet. Find the accounting boundary. Possible contributors include headers, uploads, tunneling, protocol overhead, browser subresources, retries, compression, or provider-specific billing rules.

Your provider dashboard is the authoritative source for what that provider bills. Application instrumentation is the authoritative source for what your code observed.

8. Capacity Planning Checklist

Before buying or increasing proxy traffic:

1. Define one usable business output.
2. Measure representative targets instead of using a generic page-size assumption.
3. Separate HTTP, browser, screenshot, and multi-step workflows.
4. Measure failed attempts as well as successful ones.
5. Calculate an observed attempt multiplier from a pilot.
6. Keep decimal GB and GiB definitions explicit.
7. Compare application measurements with provider billing.
8. Investigate wrong-market, challenge, parser, authentication, and transport failures separately.
9. Re-run the sample when the target, browser behavior, cadence, or extraction strategy changes.
10. Scale only after the cost per usable result is stable enough for the business case.

Common Failure Modes

The forecast is much lower than actual usage

Check whether the sample excluded redirects, browser subresources, screenshots, uploads, retries, discovery pages, or failed workflows.

Browser traffic varies wildly between runs

Look for cache differences, lazy-loaded assets, consent flows, A/B variants, session state, redirects, or target-side page changes. Measure the completed workflow rather than only the initial navigation.

Content-Length does not match what was downloaded

Do not use it as a universal transfer counter. HTTP messages can use other framing, and provider billing may include data outside the response body.

HTTP status is 200 but cost per result is poor

Add a content-level quality gate. Wrong-market pages, empty listings, soft blocks, login walls, or parser failures should not count as successful outputs.

FAQ

How many GB do I need for 100,000 pages?

There is no reliable universal number. Measure representative pages, calculate bytes per usable result, include retries and market/runs multiplicity, then extrapolate. Static HTTP pages and browser-rendered workflows can have very different traffic profiles.

Should I estimate using compressed or uncompressed page size?

Measure what your real client transfers and then reconcile it with the provider's billed usage. Do not substitute DOM size, saved HTML size, or an assumed uncompressed representation for an actual transfer measurement.

Does Playwright request.sizes() equal proxy billable traffic?

No. It gives useful HTTP request/response size information for browser requests, but it is not a universal proxy billing meter. Use it for application-side diagnostics and compare it with the provider dashboard.

Can I reduce browser traffic by blocking images?

Often, but only if the workflow remains correct. Test the same representative pages with and without blocking and validate the final business output.

What should I do after estimating GB?

If you need to translate measured traffic into cost per successful result, continue with the Residential Proxy Cost Calculator. For implementation patterns, see Web Scraping Proxy Architecture and Playwright Residential Proxy Guide.