Scrapy Framework Guide: Spiders, Middleware, Pipelines, Retries, and Proxy Routing

Scrapy is most useful when you treat it as a crawler architecture, not as a single requests.get() replacement. The Spider defines what to request and parse, the Scheduler and Downloader manage work, downloader middleware owns transport concerns such as proxy routing and retries, and item pipelines validate and persist records.

Direct answer: Keep target parsing in spiders, transport policy in downloader middleware/settings, and business validation in item pipelines. Start with conservative concurrency, enable robots handling where applicable, classify retryable failures, record crawl stats, and scale only after measuring usable output rather than raw request volume.

Scrapy's current official documentation describes it as a high-level crawling and scraping framework and documents the Engine, Scheduler, Downloader, middlewares and pipelines as separate components. See the Scrapy documentation↗ and architecture overview↗.

The mental model

Spider
  ↓ requests
Engine ↔ Scheduler
  ↓
Downloader Middleware
  ↓
Downloader
  ↓ response
Downloader Middleware
  ↓
Spider callback
  ↓ items
Item Pipeline
  ↓
Validated storage / feed

This boundary prevents one of the most common crawler problems: mixing proxy rotation, retries, HTML parsing, data validation and database writes in one callback.

1. Start with a small spider contract

import scrapy


class ProductSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/products"]

    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2,
        "DOWNLOAD_DELAY": 1.0,
    }

    def parse(self, response):
        for card in response.css("article.product"):
            detail_url = card.css("a::attr(href)").get()
            if detail_url:
                yield response.follow(detail_url, callback=self.parse_product)

    def parse_product(self, response):
        name = response.css("h1::text").get()
        price = response.css("[data-price]::attr(data-price)").get()

        yield {
            "source_url": response.url,
            "name": name.strip() if name else None,
            "price": price,
        }

The selectors above are examples for a hypothetical page. Production selectors should be derived from an authorized target and covered by fixtures or contracts.

Scrapy's generated project settings enable ROBOTSTXT_OBEY by default even though the historical global fallback differs. The current settings documentation explains this distinction. Review the target's robots rules, terms and API policies before scaling.

2. Put proxy routing in the transport layer

Scrapy's built-in HttpProxyMiddleware can use http_proxy, https_proxy and no_proxy environment variables, or a per-request proxy value in Request.meta. The current docs also note that download-handler support differs, especially for SOCKS and some HTTPS-proxy combinations. See HttpProxyMiddleware↗.

For one process-wide HTTP proxy:

export https_proxy='http://<USERNAME>:<PASSWORD>@<HOST>:<PORT>'
scrapy crawl products

For a job-specific route, set the proxy from middleware or a request factory rather than scattering it across every parser:

import os


class JobProxyMiddleware:
    def __init__(self, proxy_url: str):
        self.proxy_url = proxy_url

    @classmethod
    def from_crawler(cls, crawler):
        proxy_url = os.environ.get("SCRAPY_PROXY_URL")
        if not proxy_url:
            raise RuntimeError("SCRAPY_PROXY_URL is required")
        return cls(proxy_url)

    def process_request(self, request, spider):
        if "proxy" not in request.meta:
            request.meta["proxy"] = self.proxy_url

Activate custom downloader middleware explicitly:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.JobProxyMiddleware": 350,
}

Scrapy documents that downloader process_request() runs in increasing middleware order and process_response() in decreasing order. Middleware ordering matters when multiple components mutate requests.

Keep credentials out of logs

A proxy URL can contain credentials. Never print the full value in exception messages, stats labels or traces. Store an identifier such as proxy_route=us-rotation separately from the secret URL.

3. Understand the proxy-handler boundary

Do not assume that every Scrapy download handler supports every proxy protocol. Current Scrapy documentation states that the proxy meta key must be supported by the chosen download handler; it also notes that SOCKS proxy support is available with HttpxDownloadHandler, while other built-in handlers differ.

Verify:

1. which download handler is active;
2. whether it supports the intended proxy and destination schemes;
3. whether authentication is supported in that combination;
4. whether the exact route works against a permitted endpoint.

4. Let RetryMiddleware handle transient transport failures—but bound it

Scrapy's current RetryMiddleware is designed for temporary failures such as timeouts and selected 5xx responses. Its behavior is configurable through RETRY_TIMES, RETRY_HTTP_CODES and related settings. The current default retry count is documented as two retries in addition to the initial request.

Do not convert that default into a universal production recommendation.

RETRY_ENABLED = True
RETRY_TIMES = 2
RETRY_HTTP_CODES = [408, 429, 500, 502, 503, 504]

5. Use AutoThrottle as a control, not a permission signal

Scrapy's AutoThrottle extension↗ adjusts per-slot delay using observed latency while respecting standard concurrency and delay limits. It is useful for smoothing request pressure, but it does not decide whether a crawl is allowed.

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 0.5

Those numbers are examples. Measure the destination's documented limits and your own authorized workload.

6. Validate in an Item Pipeline

Scrapy's Item Pipeline documentation lists cleansing, validation, duplicate checks and persistence as typical pipeline responsibilities. A pipeline must return the item or raise DropItem.

from decimal import Decimal, InvalidOperation

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class ProductValidationPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)

        name = (adapter.get("name") or "").strip()
        if not name:
            raise DropItem("missing_name")

        raw_price = adapter.get("price")
        try:
            price = Decimal(str(raw_price))
        except (InvalidOperation, ValueError):
            raise DropItem("invalid_price")

        if price < 0:
            raise DropItem("negative_price")

        adapter["name"] = name
        adapter["price"] = str(price)
        return item

ITEM_PIPELINES = {
    "myproject.pipelines.ProductValidationPipeline": 300,
}

Do not silently replace invalid data with business values such as zero price or out-of-stock. A parser failure is not a market fact.

7. Use Feed Exports when custom storage code is unnecessary

Scrapy's Feed Exports↗ can write JSON, JSON Lines, CSV and other formats to supported storage backends.

FEEDS = {
    "output/%(name)s-%(time)s.jsonl": {
        "format": "jsonlines",
        "encoding": "utf-8",
        "overwrite": False,
    }
}

For very large jobs, consider batching so one interrupted upload does not make the entire run unusable.

8. Record crawl health, not just item count

Scrapy includes core statistics such as crawl start/finish time, item counts and finish reason. Add business-level counters so you can distinguish a fast crawler from a useful crawler.

Track at least:

• attempted requests;
• response status classes;
• retry reasons;
• proxy-auth failures;
• parser-invalid records;
• dropped items by reason;
• usable items;
• wrong-market records;
• bytes or proxy traffic if available;
• crawl finish reason;
• parser version.

usable_output_rate = validated_items / attempted_business_records

Do not use raw request success as the only production KPI.

9. Separate broad crawling from browser fallback

Scrapy HTTP request
  → parse required fields?
      yes → validate → pipeline
      no  → classify reason
              → browser-needed and authorized?
                    yes → browser queue
                    no  → stop / parser investigation

This keeps browser cost and complexity measurable.

10. Test parsers with saved fixtures

Keep representative HTML/JSON fixtures for normal pages, missing optional fields, missing required fields, localized variants, redirects, access/consent pages and parser regressions. Run parser tests without a network connection so transport retries cannot hide broken selectors.

Production checklist

FAQ

Should proxy rotation happen inside the spider callback?

Usually no. Keep route selection in middleware, request factories or transport configuration so parsing remains testable and independent of network policy.

Does HttpProxyMiddleware support every proxy scheme?

No. Current Scrapy docs explicitly warn that proxy support depends on the download handler. Verify the handler/proxy/destination combination you actually deploy.

Should I retry every 403 with a different proxy?

No. A 403 can represent authorization or target policy. Investigate the cause and stop when the workflow is not permitted; proxy rotation is not a permission mechanism.

Is AutoThrottle enough to make a crawler polite?

It helps regulate request timing based on latency, but you still need target-specific authorization, documented limits, robots/policy handling and explicit concurrency ceilings.

When should I use Scrapy instead of a simple HTTP client?

Use Scrapy when you benefit from a scheduler, link traversal, middleware, retry policy, pipelines, feed exports, crawl statistics and multi-page orchestration. For one or two stable API calls, a smaller HTTP client may be easier to maintain.

For proxy architecture beyond one framework, continue with Web Scraping Proxy Architecture. For Python-specific proxy setup, see Python Scraping Proxy Setup.