RAG Crawler with Proxies: Technical Architecture, Freshness, and Evidence

The system must decide what it is allowed to fetch, when content is stale, which collector is required, how to preserve evidence, how to deduplicate content, and when an index update is safe to publish.

Proxies can support geo-specific or distributed authorized collection, but they do not replace permission, source governance, or data quality controls.

Reference architecture

Every box needs an owner, a schema, retry rules, observability, and a deletion path.

1. Create a source policy registry

{
  "sourceId": "docs-example",
  "baseUrl": "https://docs.example.com/",
  "allowedPrefixes": ["https://docs.example.com/guides/"],
  "deniedPrefixes": ["https://docs.example.com/account/"],
  "robotsReviewedAt": "2026-08-01",
  "termsReviewedAt": "2026-08-01",
  "containsPersonalData": false,
  "defaultCollector": "http",
  "maxConcurrency": 2,
  "minDelayMs": 1000,
  "freshnessHours": 24,
  "owner": "knowledge-platform"
}

Robots.txt is a crawler-control mechanism, not access authorization. RFC 9309 explicitly defines the Robots Exclusion Protocol for automated clients and states that its rules are not a form of access authorization. Treat robots.txt as one input to a broader source-policy decision that also covers permission, terms, privacy, and security controls.[1]↗

2. Make the frontier stateful

A frontier record should include:

{
  "url": "https://docs.example.com/guides/setup",
  "canonicalUrl": null,
  "sourceId": "docs-example",
  "priority": 50,
  "nextFetchAt": "2026-08-07T01:00:00Z",
  "lastStatus": 200,
  "etag": "...",
  "lastModified": "...",
  "contentSha256": "...",
  "failureCount": 0,
  "leaseUntil": null
}

Use leases so two workers do not crawl the same URL simultaneously. Keep retry counters and dead-letter states explicit.

3. Prefer conditional HTTP fetching

Use ETag and Last-Modified validators when the source supports them. For GET/HEAD requests, RFC 9110 defines conditional evaluation so a matching If-None-Match or, when If-None-Match is absent, a not-modified If-Modified-Since condition can produce 304 Not Modified. This can avoid transferring an unchanged representation, but it does not prove the origin implements validators correctly.[2]↗

Pseudo-code:

async def fetch(item, client):
    headers = {}
    if item.etag:
        headers['If-None-Match'] = item.etag
    if item.last_modified:
        headers['If-Modified-Since'] = item.last_modified

    response = await client.get(item.url, headers=headers)
    if response.status_code == 304:
        return NotModified(item.url)
    response.raise_for_status()
    return Fetched(
        url=str(response.url),
        body=response.content,
        etag=response.headers.get('etag'),
        last_modified=response.headers.get('last-modified'),
    )

Do not assume every server implements validators correctly. Keep periodic full verification for critical sources.

4. Use browser fallback only when necessary

Escalate to a browser when:

• required content is rendered by JavaScript
• the page requires permitted interaction
• static HTML lacks the business content
• screenshots are needed as evidence

Do not use a browser merely because the page contains scripts. First inspect the HTTP response and any official APIs or feeds.

A browser fallback should preserve the same source ID, market profile, and evidence contract as the HTTP fetcher.

5. Put proxy selection behind a route policy

{
  "routePolicy": "direct-first",
  "allowedRouteTypes": ["direct", "residential"],
  "country": "DE",
  "sessionMode": "rotating",
  "stickyMinutes": null,
  "maxRouteAttempts": 2
}

Useful policies:

• direct-first: use direct access unless an approved geo-specific source requires a route
• fixed-market: one country and locale for a localized source
• sticky-browser-task: keep one route during a stateful browser workflow
• rotating-independent: rotate between independent, permitted page jobs

Do not rotate after every error. A 404, schema failure, or explicit access denial is not repaired by a new IP.

6. Preserve raw evidence immutably

Store:

• raw response bytes or approved normalized snapshot
• headers needed for provenance
• status and final URL
• content type and encoding
• retrieval timestamp
• collector and version
• route metadata with secrets removed
• screenshot for visual evidence when justified
• SHA-256 hash

Never use a vector database as the only copy of source evidence.

7. Canonicalize carefully

Canonicalization may include:

• normalize scheme and host casing
• remove known tracking parameters
• respect a valid canonical link when policy permits
• preserve locale and version parameters that change content
• avoid collapsing distinct pages because their titles match

Keep both requested URL and final canonical URL. Redirect history can matter during source audits.

8. Extract before chunking

Separate navigation, boilerplate, code, tables, headings, and main content. Keep document structure:

{
  "documentId": "docs-example:setup:v17",
  "sourceUrl": "https://docs.example.com/guides/setup",
  "title": "Setup Guide",
  "language": "en",
  "sections": [
    {
      "headingPath": ["Install", "Linux"],
      "text": "...",
      "sourceOffsets": [1200, 2040]
    }
  ]
}

Chunking raw HTML before extraction creates duplicate navigation and weak citations.

9. Chunk for retrieval, not arbitrary token size

A good chunk should be independently understandable and linked to source context. Use heading-aware boundaries, modest overlap, and special handling for code and tables.

Store:

• chunk ID
• document version
• heading path
• source URL
• source offsets
• content hash
• embedding model and version
• permissions label
• effective and expiry dates when relevant

10. Version embeddings and indexes

Do not overwrite the active index in place. Build a candidate index with a manifest:

{
  "indexVersion": "kb-2026-08-07-01",
  "embeddingModel": "model-name-and-version",
  "documents": 1842,
  "chunks": 11240,
  "sourceSnapshot": "snapshot-2026-08-07",
  "createdAt": "2026-08-07T00:11:00+08:00"
}

Run retrieval evaluation, then promote an alias to the candidate version. Keep rollback available.

11. Evaluate retrieval with real questions

Track:

• retrieval recall on a reviewed query set
• citation accuracy
• stale-answer rate
• permission leakage
• duplicate-source concentration
• answer abstention when evidence is missing
• latency and cost

A crawler success metric such as “10,000 pages downloaded” does not prove the RAG system answers correctly.

12. Freshness policies should depend on evidence, not a universal interval

There is no defensible single recrawl schedule for release notes, pricing, documentation, policy pages, or historical references. Set each source's freshness target from the business consequence of staleness, observed change history, source-provided validators or feeds, contractual limits, and server guidance.

Useful triggers include:

• scheduled checks for sources with a documented freshness SLO;
• conditional requests when reliable validators are available;
• event or feed-driven refresh when the source exposes one;
• manual review for high-impact policy or legal changes;
• slower verification for content that has remained stable over measured history.

Adapt intervals from observed change history, but enforce minimum delays, concurrency caps, and a maximum crawl budget so an adaptive scheduler cannot amplify load unexpectedly.

Failure taxonomy

RFC 6585 defines 429 Too Many Requests as rate limiting, permits Retry-After, and deliberately leaves the server's user-identification/counting mechanism unspecified. Rate limits can therefore be scoped by credentials, cookies, resources, or other server-defined identities rather than IP alone.[1]↗

Security and privacy

• Enforce domain and path allowlists.
• Treat web content as untrusted prompt input.
• Strip secrets before logging.
• Scan downloads and block executable content by default.
• Propagate source permissions to chunks and retrieval.
• Support deletion and re-indexing requests.
• Separate crawler credentials from answer-serving credentials.
• Audit which evidence supported each answer.

Related BytesFlows resources

• AI web data collection for RAG
• Web scraping architecture design
• Python scraping proxy setup
• AI data collection solution

Before deployment

Use the code as an implementation sketch and validate the exact client APIs, storage formats, target permissions, and rate limits in the environment where the crawler will run.