BeautifulSoup Integration
Python requests + BeautifulSoup + residential proxies: scrape without getting blocked
Add BytesFlows proxy to your requests.Session() in three lines of Python. Every request exits through a different residential IP — preventing rate limiting and IP bans on your BS4 scraping scripts.
Python requests + BeautifulSoup proxy examples
BytesFlows works directly with the requests library — no extra dependencies needed.
import requests
from bs4 import BeautifulSoup
proxies = {
'http': 'http://YOUR_USER:YOUR_PASS@p1.bytesflows.com:8001',
'https': 'http://YOUR_USER:YOUR_PASS@p1.bytesflows.com:8001',
}
response = requests.get(
'https://httpbin.org/ip',
proxies=proxies,
timeout=15,
)
print('Exit IP:', response.json()['origin'])
# Use BeautifulSoup to parse any HTML page through proxy
response = requests.get(
'https://example.com/products',
proxies=proxies,
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'},
)
soup = BeautifulSoup(response.text, 'html.parser')
products = soup.find_all('div', class_='product-card')
print(f'Found {len(products)} products')Pass the proxies dict to any requests call. For HTTPS sites, both http and https keys must be set even when using HTTP proxy protocol.
Why Python scrapers use BytesFlows with BeautifulSoup
Zero extra dependencies
The proxies parameter is built into the requests library. No proxy middleware packages, no special wrappers — just pass a dict with proxy URLs and you're done.
Rotating mode by default
BytesFlows rotates the exit IP on each new TCP connection. requests.get() calls each get a fresh residential IP — perfect for scraping large product catalogs or SERP pages where per-URL IP diversity is critical.
Works with httpx and aiohttp
Same proxy URL format works with modern async libraries. Use BytesFlows with httpx for async scraping, or aiohttp for high-concurrency data collection — same credentials, same rotating residential IPs.