How to Avoid IP Bans in Web Scraping: Rate, Sessions, Retries, and Stop Conditions

An IP ban is usually a symptom, not a root cause. The underlying issue may be excessive concurrency, repeated retries, malformed authentication, an unstable session, a disallowed crawler path, or a target policy that does not permit the automated workflow.

Start with permission and crawler scope

Before tuning retries, define what the crawler is allowed to fetch.

The Robots Exclusion Protocol is standardized in RFC 9309↗. It gives service owners a way to communicate crawler access preferences through /robots.txt; the RFC also makes clear that robots rules are not access authorization.

Your job should still document:

• authorization basis
• target hosts and allowed paths
• request-rate limits
• personal-data handling
• retention policy
• stop conditions

If an API or licensed feed can answer the business question, prefer it over a heavier browser path.

Control concurrency before adding more proxies

A common failure pattern is multiplying workers while keeping no per-target budget.

Use a target-level semaphore:

import asyncio

TARGET_CONCURRENCY = 4
semaphore = asyncio.Semaphore(TARGET_CONCURRENCY)

async def bounded(fetch, url):
    async with semaphore:
        return await fetch(url)

The number 4 is an example, not a universal safe limit. Start conservatively and increase only when the target's terms, your authorization and observed service behavior support it.

Add rate control, not just a worker limit

Concurrency limits cap simultaneous requests but do not guarantee a stable request rate. Use a scheduler or token bucket when the target needs a predictable budget.

Track rate per target, not only globally. A crawler touching ten domains should not accidentally send the entire global allowance to one host.

Classify failures before retrying

Do not send every non-200 response into the same retry loop.

Do not treat a challenge page returned with HTTP 200 as a successful scrape.

Use exponential backoff with a total budget

import asyncio
import random

async def backoff(attempt: int) -> None:
    cap = min(2 ** attempt, 30)
    await asyncio.sleep(random.uniform(0, cap))

Also enforce:

• maximum attempts
• maximum wall-clock time
• maximum bytes
• maximum cost per logical job
• maximum redirects

A retry policy without a total budget can turn one blocked URL into a traffic amplifier.

Respect server retry guidance when present

If a service communicates a retry interval, your scheduler should honor it rather than immediately switching identity and trying again. A new IP does not erase the server's stated rate limit or your obligation to respect the service's rules.

Decide when a session should stay sticky

Rotating every request is not automatically safer.

Sticky sessions are often more coherent for:

• login or authorized account workflows
• pagination with server-side state
• carts or multi-step forms
• browser journeys

Rotation can fit independent public-page observations when the workflow is authorized and route distribution is part of the design.

Changing IP midway through a stateful workflow can create more anomalies and more failures.

Avoid retrying a bad credential as a new identity

Separate errors into at least four classes:

type RetryDecision =
  | 'stop'
  | 'backoff'
  | 'new-route'
  | 'parser-review';

Examples:

• 407 → stop
• explicit denial → stop
• temporary connect reset → maybe new-route
• 429 → backoff
• DOM schema mismatch → parser-review

This prevents a route pool from hiding application bugs.

Cache and deduplicate work

Repeatedly fetching the same URL is one of the easiest sources of avoidable traffic.

Use:

• URL normalization
• crawl frontier deduplication
• content hashes
• conditional requests where supported
• reasonable freshness TTLs
• change detection before full browser rendering

A cheaper request is often the request you never send.

Prefer HTTP before a browser when the data allows it

Browser automation may load scripts, images, analytics and API calls far beyond the one field you need.

A sensible ladder is:

approved API/feed
    ↓
HTTP document or JSON
    ↓
hydration / embedded state
    ↓
browser only when rendering or interaction is required

This reduces bandwidth, concurrency pressure and failure surface.

Record evidence for blocks

For a blocked or suspicious result, log enough to diagnose it:

{
  "jobId": "crawl-001",
  "targetHost": "example.com",
  "attempt": 2,
  "status": 429,
  "pageClass": "rate_limited",
  "proxyMode": "sticky",
  "durationMs": 2100,
  "decision": "backoff"
}

Do not log passwords, reusable session tokens or unnecessary personal data.

For browser jobs, capture a trace or sanitized snapshot when permitted. Review evidence retention because pages can contain sensitive data.

Define explicit stop conditions

A production crawler should stop or quarantine work when:

• credentials are rejected
• authorization is uncertain
• robots or contractual rules prohibit the path
• the target returns repeated explicit denial
• a rate-limit threshold is exceeded
• the content becomes a login, challenge or consent wall outside your approved flow
• the parser can no longer prove the expected business output

These rules are more valuable than “try another IP until it works.”

Metrics that reveal unhealthy crawling

Monitor:

• valid business outputs / attempts
• 403 and 429 rate
• transport-failure rate
• retries per valid output
• bytes per valid output
• median and p95 latency
• duplicate-fetch rate
• challenge-page rate
• cost per valid output

If adding proxies increases attempts but not valid output, the problem is not proxy-pool size.

Where proxy rotation belongs

A proxy pool can provide route diversity, geography and failure isolation. It should not be used to circumvent explicit access restrictions.

For route selection, keep policy separate from mechanism:

Can this URL be fetched?
        ↓
Which session mode does the task need?
        ↓
Which healthy authorized route fits that session?

That ordering prevents routing logic from becoming the policy engine.

Production checklist

1. Authorization and allowed paths are documented.
2. /robots.txt is evaluated where applicable.
3. Per-target concurrency is bounded.
4. Rate is explicitly controlled.
5. All network operations have timeouts.
6. 401/403/407/429 are classified separately.
7. Retries have jitter and a total budget.
8. Stateful jobs do not rotate accidentally.
9. Duplicate URLs are suppressed.
10. Browser rendering is used only when necessary.
11. Evidence exists for invalid outputs.
12. Explicit denials trigger a stop/review path.

Related BytesFlows guides

• Web Scraping Proxy Architecture
• Proxy Rotation Strategy
• IP Reputation Check
• Why Residential Proxies Work Better for Some Scraping Workloads
• Web scraping proxies

When to stop

These controls can reduce avoidable failures, but no route guarantees that a target will accept the workflow. Validate rate, session behavior, and retry rules against the exact target you are authorized to access, and stop when access rules or an explicit denial require it.