SERP Scraping Proxy QA: Schema, Geo Validation & Data Quality Gates

A SERP scraping proxy setup is only useful if the output can be compared over time. The core engineering problem is therefore not “did the request return HTML?” but “can we prove which query, market, locale, device, route, parser version, and evidence produced this rank record?”

This guide focuses on that data-quality contract. For collection architecture, browser automation, and retry design, use SERP Scraping with Residential Proxies. For recurring rank-tracking workflow design, see Proxies for Rank Tracking.

Scope: Use the techniques below only on sources you are authorized to access and in ways consistent with applicable terms, privacy obligations, and rate limits. A proxy changes the network route and exit IP; it does not make automation authorized, guarantee a particular SERP, or bypass security controls.

What a SERP data-quality contract should prove

Before storing a rank observation, the record should answer five questions:

1. What was requested? Query text, market, locale, device class, and requested time.
2. What route was actually observed? Proxy mode plus independently observed country/region/city when location matters.
3. What came back? HTTP outcome, result count, result URLs/titles, and any challenge or interstitial signal your parser can identify reliably.
4. Which parser produced the record? Parser version, schema version, and extraction timestamp.
5. Can the result be audited? Preserve an evidence pointer such as a content hash, sanitized response artifact, or screenshot reference according to your retention policy.

Google documents that serving can depend on signals including a user's location, language, and device. That is enough reason to store these dimensions explicitly instead of assuming that the proxy country alone defines the search context.[1]↗

A practical JSON Schema for rank observations

JSON Schema Draft 2020-12 is a current published specification and is suitable for validating a stable record contract.[2]↗

The example below is intentionally provider-neutral. Values such as proxy endpoint syntax, session tokens, and supported targeting fields should come from the provider's current dashboard or documentation rather than being hard-coded from an old article.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.com/schemas/serp-record-v1.json",
  "title": "SerpRecordV1",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "snapshot_id",
    "query",
    "requested_context",
    "observed_context",
    "fetch",
    "results",
    "parser_version",
    "captured_at"
  ],
  "properties": {
    "snapshot_id": {"type": "string", "minLength": 1},
    "query": {"type": "string", "minLength": 1},
    "requested_context": {
      "type": "object",
      "additionalProperties": false,
      "required": ["country", "language", "device"],
      "properties": {
        "country": {"type": "string", "pattern": "^[A-Z]{2}$"},
        "region": {"type": ["string", "null"]},
        "city": {"type": ["string", "null"]},
        "language": {"type": "string", "minLength": 2},
        "device": {"type": "string", "enum": ["desktop", "mobile", "tablet"]}
      }
    },
    "observed_context": {
      "type": "object",
      "additionalProperties": false,
      "required": ["exit_ip", "country"],
      "properties": {
        "exit_ip": {"type": "string", "minLength": 1},
        "country": {"type": "string", "pattern": "^[A-Z]{2}$"},
        "region": {"type": ["string", "null"]},
        "city": {"type": ["string", "null"]}
      }
    },
    "fetch": {
      "type": "object",
      "additionalProperties": false,
      "required": ["status_code", "duration_ms", "proxy_mode"],
      "properties": {
        "status_code": {"type": "integer", "minimum": 100, "maximum": 599},
        "duration_ms": {"type": "integer", "minimum": 0},
        "proxy_mode": {"type": "string", "enum": ["rotation", "sticky", "direct"]},
        "retry_count": {"type": "integer", "minimum": 0}
      }
    },
    "results": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["rank", "url", "title"],
        "properties": {
          "rank": {"type": "integer", "minimum": 1},
          "url": {"type": "string", "format": "uri"},
          "title": {"type": "string"}
        }
      }
    },
    "parser_version": {"type": "string", "minLength": 1},
    "evidence_ref": {"type": ["string", "null"]},
    "captured_at": {"type": "string", "format": "date-time"}
  }
}

Treat format validation deliberately

The Python jsonschema library supports Draft 2020-12 validators. Its documentation also notes that format checking is not automatically enforced just because a schema contains format keywords; enable a format checker when URI/date-time validation is part of your contract.[3]↗

Validate JSONL records before they enter analytics storage

Keep transport and parsing separate from the quality gate. This makes the validator easy to run in CI, replay jobs, or quarantine processing without making another network request.

Install:

python -m pip install 'jsonschema[format]'

Save the schema as serp-record.schema.json, then validate a JSONL file with:

import json
import sys
from pathlib import Path

from jsonschema import Draft202012Validator, FormatChecker


def load_json(path: Path) -> dict:
    with path.open("r", encoding="utf-8") as fh:
        return json.load(fh)


def validate_jsonl(schema_path: Path, records_path: Path) -> int:
    schema = load_json(schema_path)
    Draft202012Validator.check_schema(schema)
    validator = Draft202012Validator(schema, format_checker=FormatChecker())

    failures = 0
    with records_path.open("r", encoding="utf-8") as fh:
        for line_number, line in enumerate(fh, start=1):
            if not line.strip():
                continue

            try:
                record = json.loads(line)
            except json.JSONDecodeError as exc:
                failures += 1
                print(f"line={line_number} invalid_json={exc}", file=sys.stderr)
                continue

            errors = sorted(validator.iter_errors(record), key=lambda e: list(e.path))
            if not errors:
                continue

            failures += 1
            print(f"line={line_number} validation_failed", file=sys.stderr)
            for error in errors:
                field = ".".join(str(part) for part in error.path) or "<root>"
                print(f"  {field}: {error.message}", file=sys.stderr)

    return failures


if __name__ == "__main__":
    failed = validate_jsonl(
        Path("serp-record.schema.json"),
        Path("serp-records.jsonl"),
    )
    raise SystemExit(1 if failed else 0)

A non-zero exit code makes the same validator usable as a CI or batch-ingestion gate.

Verify requested GEO and observed GEO separately

Do not write country=US merely because the credential requested the United States. Store both:

• requested_context.country: what the job asked the proxy/provider to route;
• observed_context.country: what an independent IP/geo check reported for the actual exit;
• the source and timestamp of that geo observation when it matters to your audit trail.

IP geolocation is an estimate, especially at city level. Treat city mismatches as evidence to investigate, not proof that one system is “wrong.” If city precision is a contractual requirement, define the accepted provider/source and tolerance in advance.

For location-sensitive search work, also keep language and device as separate dimensions. Google states that search relevance can include location, language, and device, so identical queries do not imply identical result contexts.[1]↗

Use a QA gate, not a single qa_passed guess

A useful gate records why a row passed or failed. For example:

Distinguish 407, 429, and target-site failures

407 Proxy Authentication Required is defined by HTTP as a proxy authentication challenge. Treat it as a proxy configuration/authentication problem and stop target-site retries until credentials or proxy configuration are fixed.[4]↗

429 Too Many Requests indicates rate limiting and may include Retry-After. RFC 6585 explicitly does not define how a server identifies the user or counts requests; it may be based on credentials, cookies, a resource, or other scope. Therefore, changing an IP is not a general solution to 429 responses.[5]↗

For 403, challenge pages, consent pages, or changed markup, inspect the actual response and your authorization boundary. Do not label every non-200 result as “bad proxy.”

HTTPX proxy configuration: keep it explicit

If a QA probe itself needs HTTPX, configure the proxy on the client (or a top-level request) using HTTPX's documented proxy interface. HTTPX currently documents httpx.Client(proxy=...) / httpx.AsyncClient(proxy=...) and mounts for advanced routing.[6]↗

import asyncio
import os
import httpx


async def probe_authorized_endpoint() -> None:
    proxy_url = os.environ["PROXY_URL"]
    target_url = os.environ["AUTHORIZED_TEST_URL"]

    timeout = httpx.Timeout(20.0, connect=10.0)
    async with httpx.AsyncClient(proxy=proxy_url, timeout=timeout) as client:
        response = await client.get(target_url)
        response.raise_for_status()
        print(response.status_code)


if __name__ == "__main__":
    asyncio.run(probe_authorized_endpoint())

PROXY_URL is an example environment variable. Use the endpoint, authentication format, GEO syntax, and session controls shown by your current BytesFlows dashboard. Do not copy a hostname, port, or username grammar from an old article and assume it is permanent.

Define retries as a data-quality rule

Retries change the experiment. If the first request fails and the second succeeds through a different route, the final rank record no longer describes a single attempt.

Store at least:

• attempt number;
• route/session identifier that is safe to log;
• status class or failure class;
• start/end timestamp;
• whether the route changed;
• final accepted record ID.

Set a finite attempt budget. Respect explicit Retry-After instructions and provider/target limits. Stop when authentication fails, authorization is unclear, the source asks automation to stop, or retries would only amplify the same failure.

Minimal validation checklist before scaling

FAQ

Does a residential proxy guarantee accurate local SERP data?

No. A residential exit can provide a requested network location, but search results can depend on additional signals and can change over time. Validate the actual exit context and store language, device, timestamp, parser version, and other dimensions required by your measurement design.

Should every failed scrape be retried with a new IP?

No. First classify the failure. A 407 is a proxy-authentication problem; a 429 is rate limiting whose scope is not necessarily IP-based; a parser failure requires parser evidence; and a policy or authorization stop should not be retried.

Should failed schema records be written to the main rank table?

Usually no. Put them in a quarantine path with the validation errors and enough permitted evidence to reproduce the parser issue. Keep the primary analytics table limited to records that satisfy the declared contract.

Is city-level IP geolocation exact?

No. Treat IP geolocation as an estimate and define which geo database or provider observation is authoritative for your own QA process. Country-level and city-level confidence should not be treated as interchangeable.

How is this different from the main SERP scraping article?

This page is the record contract and QA gate: schema versioning, requested-vs-observed context, validation, failure classification, and ingestion rules. The main SERP Scraping with Residential Proxies article covers collection workflow and proxy usage more broadly.

References

• JSON Schema Draft 2020-12 specification.[2]↗
• Python jsonschema validator documentation.[3]↗
• HTTPX proxy documentation.[6]↗
• RFC 9110, HTTP semantics and 407 proxy authentication.[4]↗
• RFC 6585, 429 Too Many Requests.[5]↗
• Google Search Central, how serving can vary with factors including location, language, and device.[1]↗