Public Web Data Collection When APIs Are Unavailable: Structured Extraction, Change Detection, and Evidence

A public website can be the only authoritative source for a business dataset even when no official API exists, the API is incomplete, or the API does not expose the market-specific state your workflow needs. In that situation, the production problem is not “how do I scrape HTML?” It is how to build a controlled collection system that turns authorized public pages into versioned, explainable records without confusing transport failures, parser drift, wrong-market responses, and real business changes.

Direct answer: Use the least complex approved source first, put proxy routing in the transport layer rather than the parser, validate geography and business fields before storage, preserve evidence for every accepted observation, and treat change detection as a separate stateful system. Rotation helps independent observations; sticky sessions are only for bounded workflows that genuinely require continuity. Neither mode guarantees access or bypasses target controls.

A. Business definition and success criteria

Who this architecture is for

This pattern fits teams building recurring public-web datasets for market intelligence, RAG ingestion, product catalogs, public directories, research, pricing, availability, local content, or operational monitoring when the desired information is publicly accessible but there is no suitable official API.

A production job should start with a collection contract:

{
  "dataset": "example-public-catalog",
  "target_url": "https://example.com/items/123",
  "market": {
    "country": "US",
    "region": null,
    "city": null,
    "language": "en-US",
    "currency": "USD"
  },
  "refresh_policy": "scheduled",
  "expected_record_type": "product",
  "evidence_required": true,
  "allowed_source_layers": ["official_json", "jsonld", "embedded_json", "html", "browser"],
  "retention_class": "business-audit"
}

The input is an approved target set plus market, refresh, schema and retention rules. The output is not merely HTML: it is a normalized record, collection metadata, source URL, observed geography, timestamp, parser version, content hash, evidence pointer and final status.

Success means:

• the correct approved target was collected;
• the observed market matches the requested market when geography matters;
• required business fields pass schema validation;
• duplicate observations are idempotently collapsed;
• meaningful changes are distinguishable from parser or transport failures;
• evidence exists for records that drive decisions;
• failures are classified and observable rather than converted into false business facts.

Appropriate and inappropriate uses

Appropriate uses include recurring collection of public information that your organization is authorized to access, especially where content differs by geography or where an API lacks the required fields.

Do not use this architecture to bypass login requirements, paywalls, access controls, CAPTCHAs, platform security mechanisms, or contractual restrictions. Do not introduce a proxy merely because a parser is broken. If the same page is public and globally identical, ordinary direct collection may be simpler and more appropriate.

B. End-to-end system architecture

The customer owns scheduling, parsing, validation, storage, evidence policy and monitoring. BytesFlows provides the network routing layer used when the collection contract requires a particular network geography or controlled proxy session behavior.

Extraction ladder

Use the least complex approved data source that contains the required business fields:

Official public JSON/API
        ↓ unavailable or insufficient
JSON-LD / embedded structured JSON
        ↓ unavailable
Stable server-rendered HTML
        ↓ unavailable
Documented client-side public data source
        ↓ unavailable
Browser-rendered public DOM

This ordering keeps browsers and stateful sessions out of jobs that do not need them. It also makes parser testing easier because saved fixtures can be replayed without network access.

C. Dynamic proxy strategy

Proxy behavior should be derived from the business observation, not from a blanket “rotate every request” rule.

GEO rules

Treat requested and observed geography as separate fields:

{
  "requested_geo": {"country": "DE", "region": "BE", "city": "Berlin"},
  "observed_geo": {"country": "DE", "region": "BE", "city": "Berlin"},
  "geo_validation": "pass"
}

If country, region or city is part of the job contract, every specified level must pass validation before market-sensitive data is accepted. Never silently degrade a city request into country-only data.

Session lifecycle

For sticky workflows, bind the session to a business run rather than a human-readable target:

session_key = dataset_id + market_id + workflow_id + run_id

Create the session at workflow start, reuse it only for the bounded workflow, and release it on completion, cancellation, expiry or a transport condition that invalidates the route. Do not reuse one sticky session across unrelated customers, markets or long-running dataset partitions.

Concurrency should be limited by both target domain and account capacity. The correct number is workload-specific; this article does not claim a universal thread count.

D. Request and task scheduling design

A reliable collector needs explicit queue semantics.

Task identity and idempotency

Use an idempotency key derived from the business observation window:

sha256(dataset_id | canonical_url | market_id | scheduled_window | parser_version)

Before enqueueing, check whether the observation already reached a terminal state. On worker restart, the same key can be safely retried without creating duplicate business records.

Domain-level controls

Maintain per-domain settings for:

• maximum in-flight work;
• minimum spacing between requests where required;
• retry budget;
• circuit-breaker state;
• browser allowance;
• robots and policy decision;
• parser version;
• evidence retention class.

State machine

Failure decision table

Backoff and circuit breakers

Use exponential or decorrelated backoff with jitter for retryable transport conditions. Do not retry every failure class. Open a per-domain or per-parser circuit when systemic failures exceed the operating threshold defined by your team. The threshold should be derived from your own error budget rather than copied from a generic example.

Cancellation must propagate from campaign or dataset scope down to queued tasks and active browser contexts so an operator can stop collection quickly.

E. Runnable implementation

The following Python example is a coherent reference worker for approved HTTP collection. It uses environment variables for proxy credentials, bounded timeouts, response classification, structured extraction, schema validation and evidence metadata.

This code was not executed in this content run and must not be treated as a tested BytesFlows benchmark. Replace the placeholder host, credentials, target URLs and account-specific routing format with the configuration documented for your account.

from __future__ import annotations

import hashlib
import json
import os
import random
import time
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from typing import Any

import requests
from bs4 import BeautifulSoup

PROXY_HOST = os.environ.get("BYTESFLOWS_PROXY_HOST", "proxy.example.invalid:8000")
PROXY_USER = os.environ.get("BYTESFLOWS_PROXY_USER", "ACCOUNT_USER_PLACEHOLDER")
PROXY_PASS = os.environ.get("BYTESFLOWS_PROXY_PASS", "ACCOUNT_PASSWORD_PLACEHOLDER")


@dataclass(frozen=True)
class Market:
    country: str
    region: str | None = None
    city: str | None = None


@dataclass(frozen=True)
class ProductRecord:
    source_url: str
    external_id: str
    name: str
    currency: str
    price: Decimal
    availability: str | None


class CollectionError(Exception):
    category = "collection_error"


class ProxyAuthError(CollectionError):
    category = "proxy_auth_error"


class TransportError(CollectionError):
    category = "proxy_transport_error"


class TargetHTTPError(CollectionError):
    category = "target_http_error"


class ParserError(CollectionError):
    category = "parser_error"


def proxy_url() -> str:
    if "PLACEHOLDER" in PROXY_USER or "PLACEHOLDER" in PROXY_PASS:
        raise RuntimeError("Set proxy credentials through environment variables before running")
    return f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}"


def request_html(url: str) -> requests.Response:
    proxy = proxy_url()
    try:
        response = requests.get(
            url,
            proxies={"http": proxy, "https": proxy},
            headers={"User-Agent": "authorized-public-data-collector/1.0"},
            timeout=(5, 20),
        )
    except requests.exceptions.ProxyError as exc:
        raise TransportError(str(exc)) from exc
    except requests.exceptions.RequestException as exc:
        raise TransportError(str(exc)) from exc

    if response.status_code == 407:
        raise ProxyAuthError("Proxy authentication failed")
    if response.status_code >= 400:
        raise TargetHTTPError(f"Target returned HTTP {response.status_code}")
    if "text/html" not in response.headers.get("content-type", "").lower():
        raise ParserError("Expected HTML response")
    return response


def iter_jsonld(html: str):
    soup = BeautifulSoup(html, "html.parser")
    for script in soup.find_all("script", attrs={"type": "application/ld+json"}):
        raw = script.string or script.get_text()
        if not raw or not raw.strip():
            continue
        try:
            value = json.loads(raw)
        except json.JSONDecodeError:
            continue
        if isinstance(value, dict) and isinstance(value.get("@graph"), list):
            for node in value["@graph"]:
                if isinstance(node, dict):
                    yield node
        elif isinstance(value, dict):
            yield value
        elif isinstance(value, list):
            yield from (node for node in value if isinstance(node, dict))


def parse_product(url: str, html: str) -> ProductRecord:
    node: dict[str, Any] | None = None
    for candidate in iter_jsonld(html):
        node_type = candidate.get("@type")
        if node_type == "Product" or (isinstance(node_type, list) and "Product" in node_type):
            node = candidate
            break
    if node is None:
        raise ParserError("No Product JSON-LD record found")

    offers = node.get("offers")
    if isinstance(offers, list):
        offers = offers[0] if offers else None
    if not isinstance(offers, dict):
        raise ParserError("Product offers missing")

    try:
        price = Decimal(str(offers["price"]))
    except (KeyError, InvalidOperation, ValueError) as exc:
        raise ParserError("Invalid price") from exc

    currency = str(offers.get("priceCurrency", "")).upper()
    if len(currency) != 3:
        raise ParserError("Invalid currency")

    external_id = str(node.get("sku") or node.get("productID") or "").strip()
    name = str(node.get("name") or "").strip()
    if not external_id or not name:
        raise ParserError("Required identity fields missing")

    return ProductRecord(
        source_url=url,
        external_id=external_id,
        name=name,
        currency=currency,
        price=price,
        availability=str(offers.get("availability") or "") or None,
    )


def content_hash(body: bytes) -> str:
    return hashlib.sha256(body).hexdigest()


def collect(url: str, market: Market, max_attempts: int = 3) -> dict[str, Any]:
    last_error: CollectionError | None = None

    for attempt in range(1, max_attempts + 1):
        try:
            response = request_html(url)
            record = parse_product(response.url, response.text)
            captured_at = datetime.now(timezone.utc).isoformat()

            return {
                "status": "valid_observation",
                "captured_at": captured_at,
                "requested_geo": asdict(market),
                "observed_geo": None,  # populate from your approved GEO validation step
                "geo_validation": "pending",
                "parser_version": "product-jsonld-v1",
                "http_status": response.status_code,
                "source_url": response.url,
                "content_sha256": content_hash(response.content),
                "record": {
                    **asdict(record),
                    "price": str(record.price),
                },
            }
        except ProxyAuthError:
            raise
        except ParserError:
            raise
        except (TransportError, TargetHTTPError) as exc:
            last_error = exc
            if attempt >= max_attempts:
                break
            time.sleep(min(8.0, (2 ** (attempt - 1)) + random.random()))

    assert last_error is not None
    raise last_error


if __name__ == "__main__":
    target = "https://PUBLIC_TARGET_PLACEHOLDER.example/item/123"
    result = collect(target, Market(country="US"))
    print(json.dumps(result, indent=2, default=str))

Add browser fallback only when required

If the required field is generated only after client-side execution, use Playwright as a separate worker class rather than mixing browser logic into the HTTP parser. Preserve the same task ID, market contract and evidence schema. A browser fallback should still stop at explicit access controls; it is not a challenge-bypass layer.

F. Data quality and evidence

Separate the observation from the resolved business entity.

A useful observation schema is:

{
  "observation_id": "obs_...",
  "dataset_id": "catalog_public_v1",
  "entity_key": "merchant|market|external_id",
  "source_url": "https://example.com/items/123",
  "captured_at": "2026-08-15T01:00:00Z",
  "requested_geo": {"country": "US"},
  "observed_geo": {"country": "US"},
  "exit_ip_evidence_id": "geo_...",
  "http_status": 200,
  "source_layer": "jsonld",
  "parser_version": "product-jsonld-v1",
  "content_sha256": "...",
  "record_sha256": "...",
  "evidence_uri": "object://evidence/...",
  "validation": "pass"
}

Deduplication

Deduplicate at two levels:

1. Task dedupe prevents the same scheduled observation from running twice.
2. Record dedupe identifies semantically identical normalized records even if the raw HTML changed for unrelated reasons.

Do not use the full HTML hash alone as the business change signal. Advertising slots, recommendation widgets, timestamps or unrelated markup can change while the product record stays the same.

Change detection

Compare normalized, versioned business fields:

previous valid observation
        ↓
entity resolution
        ↓
field-level diff
        ↓
classification
  ├─ no business change
  ├─ expected change
  ├─ anomalous change
  ├─ source missing
  └─ insufficient evidence

For every reported change, store the before/after observation IDs, parser versions, timestamps and evidence pointers. If the parser version changes at the same time as the data, classify the result cautiously until replay or review distinguishes parser behavior from a real source change.

Empty and partial results

An empty page is not automatically an empty dataset. Require an explicit parser signal that distinguishes:

• valid empty result;
• wrong market;
• consent/interstitial state;
• parser failure;
• partial response;
• removed entity.

Only a valid empty observation should update the business state to “no items” or “not found.”

G. Production reliability

Track metrics at dataset, domain, market, parser and proxy-route levels.

Recommended metrics include:

• task completion rate;
• valid observation rate;
• transport retry rate;
• proxy authentication failures;
• target 4xx/5xx distribution;
• parser failure rate by parser version;
• GEO mismatch rate;
• browser fallback rate;
• evidence-write failures;
• duplicate suppression rate;
• change-event rate;
• anomalous-field rate;
• queue age and worker saturation;
• p50/p95 collection latency measured by your own system.

Do not set alert thresholds from this article. Establish them from your workload and error budget.

Structured logs

Every terminal task should log fields such as:

{
  "task_id": "task_...",
  "dataset_id": "catalog_public_v1",
  "domain": "example.com",
  "market_id": "US",
  "session_mode": "rotation",
  "attempt": 2,
  "parser_version": "product-jsonld-v1",
  "result_class": "parser_error",
  "http_status": 200,
  "geo_validation": "pass",
  "evidence_id": "ev_...",
  "credential_fields_redacted": true
}

Failure samples and human review

Retain a bounded sample of failures with sanitized evidence. Route systematic parser drift, unexplained market mismatch and high-impact anomalous changes to human review. Do not keep every response indefinitely just because storage is cheap; retention must follow a documented business and privacy purpose.

H. Security, privacy and compliance

Before a target enters the queue:

• confirm the data is public and the intended collection is authorized;
• review applicable terms and robots directives;
• identify whether personal data is present and whether it is necessary;
• minimize fields collected and retained;
• keep proxy passwords, cookies and tokens out of source code and logs;
• restrict evidence-store access;
• define retention and deletion schedules;
• stop collection when permissions, policies or access conditions change.

Do not provide automation to bypass login, account permissions, paywalls, CAPTCHAs or other security mechanisms. A proxy changes the network path; it does not grant permission to access content.

I. Launch checklist and scaling path

Development

Pre-production

Production

Scaling path

Start with one dataset, a small target set and one or two markets. As volume grows, split components by responsibility: queue partitions, HTTP worker pools, browser worker pools, evidence object storage, entity-resolution service, change-event pipeline and independent circuit breakers per noisy domain.

Scale geography only after the evidence model can prove which market produced each record. Scale concurrency only after target-domain controls, retries and cancellation are observable. Scale browser use last because it increases cost and operational surface area.

Rollback and stop conditions

Stop or roll back a parser release when required-field failures increase systemically, when new records cannot be reconciled with saved fixtures, or when evidence cannot prove reported changes. Pause a target when policy authorization changes, GEO validation repeatedly fails, or the target begins returning access controls your system is not permitted to automate around.

J. Conversion design: validate the route before scaling

Before connecting a recurring dataset to any proxy provider, verify the network path you intend to depend on: confirm the observed exit, requested geography, account configuration and target behavior with a small controlled sample. The goal is to separate network evidence from parser and business-data evidence before production volume makes diagnosis harder.

Primary CTA: Run a Proxy Test

Supporting references inside BytesFlows:

• Residential proxy locations — confirm which markets you need before designing the collection matrix.
• Current pricing — estimate the account plan only after target count, refresh frequency and market scope are known.
• RAG crawler architecture — extend this evidence model into freshness-controlled knowledge ingestion.
• Web scraping proxy architecture — separate session routing, retries and observability from extraction logic.

The production goal is not to collect the maximum number of pages. It is to maintain a dataset where every accepted record can be traced to a target, time, market, parser version and evidence artifact—and where the system can explain why a change is real rather than merely the result of a failed request or broken parser.