Scrapy proxy configuration examples
BytesFlows works with Scrapy's built-in HttpProxyMiddleware and popular third-party rotation libraries.
# settings.py
# Enable HTTP proxy middleware
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}
# BytesFlows proxy credentials
# Using rotating mode: each request gets a different residential IP
HTTP_PROXY = 'http://YOUR_USER:YOUR_PASS@p1.bytesflows.com:8001'
# For SOCKS5 instead of HTTP:
# HTTP_PROXY = 'socks5://YOUR_USER:YOUR_PASS@p1.bytesflows.com:1080'
# Optional: Retry on proxy errors
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]Replace YOUR_USER and YOUR_PASS with credentials from BytesFlows Dashboard → Proxies → Residential → Sub-users. The rotating mode automatically selects a different exit IP for each request.
How Scrapy routes requests through BytesFlows
Scrapy's HttpProxyMiddleware intercepts outgoing download requests and forwards them through the configured proxy.
Scrapy spider
Spider generates Request objects. HttpProxyMiddleware reads the HTTP_PROXY setting and attaches proxy credentials to each request.
Downloader middleware
The middleware stack processes requests in priority order. HttpProxyMiddleware (priority 110) adds the Proxy-Authorization header.
BytesFlows gateway
The gateway validates credentials, selects a residential exit node based on any geo parameters in the username.
Residential exit
Request exits through a real home broadband IP. The target server sees a legitimate residential visitor, not a datacenter crawler.
BytesFlows supports both rotating and sticky session modes. Rotating mode (default) changes IP per request — ideal for bulk scraping. Sticky mode maintains IP for multi-step flows.
Built for production Scrapy pipelines
HttpProxyMiddleware compatibility
No third-party library needed. BytesFlows works with Scrapy's built-in HttpProxyMiddleware via a single setting in settings.py. SOCKS5 and HTTP both supported.
Works with scrapy-rotating-proxies
For proxy list rotation, add BytesFlows endpoints with different sub-user credentials to the ROTATING_PROXY_LIST. Each item gets a different residential exit IP for true rotation.
Retry-compatible proxy pool
BytesFlows gracefully handles retries — if an exit node is slow, the gateway automatically switches to a healthy node. Combine with Scrapy's RETRY_TIMES for resilient production pipelines.