HTTP Header Checker for Proxy Diagnostics

A header check is evidence, not a verdict

HTTP headers are one of the fastest ways to diagnose redirects, proxy authentication, rate limits, caching, content negotiation, and response mismatches. They are also easy to overinterpret.

A header checker can answer questions such as:

Did the server redirect the request?
Did an intermediary add a Via or Forwarded field?
Is the response cacheable?
Is the server asking the client to retry later?
Did content negotiation change the representation?
Is the response compressed?
Did the proxy reject authentication with 407?

It cannot, by itself, prove IP reputation, geolocation accuracy, TLS/browser fingerprint equivalence, account health, or whether the returned body contains usable data.

Start with the right curl command

For response headers only:

bash

curl --silent --show-error --head https://example.com/

For response headers and a normal GET request:

bash

curl --silent --show-error \
  --dump-header - \
  --output /dev/null \
  https://example.com/

For connection and protocol diagnostics:

bash

curl --verbose \
  --output /dev/null \
  https://example.com/

Do not paste verbose output into a public ticket before redacting credentials, cookies, signed query parameters, and authorization data.

HEAD is not the same test as GET

RFC 9110 defines HEAD as the same semantics as GET except that the server does not send response content. It also allows a server to omit fields whose values are determined only while generating the content.

That means curl -I is excellent for a lightweight metadata check, but it is not a perfect substitute for the actual GET path. If a production failure happens on GET, reproduce it with GET and capture its headers.

Reference: RFC 9110 §9.3.2 — HEAD↗

A repeatable proxy-header test

Use environment variables so credentials are not embedded in shell history or documentation:

bash

export PROXY_HOST='proxy.example.net:9000'
export PROXY_USER='customer-example'
export PROXY_PASS='replace-me'

curl --fail-with-body \
  --silent --show-error \
  --proxy "http://${PROXY_HOST}" \
  --proxy-user "${PROXY_USER}:${PROXY_PASS}" \
  --connect-timeout 10 \
  --max-time 30 \
  --dump-header /tmp/response.headers \
  --output /tmp/response.body \
  https://example.com/

The timeout values are examples, not BytesFlows defaults. Inspect /tmp/response.headers locally and remove sensitive values before sharing it.

Headers worth checking

Header / signal	What it can tell you	Common mistake
`Location`	Redirect destination	Assuming the first URL is the final content URL
`Via`	Presence of HTTP intermediaries that choose to identify themselves	Assuming absence means no proxy exists
`Forwarded`	Optional standardized forwarding information	Treating it as guaranteed or safe to expose publicly
`Retry-After`	Requested delay before a follow-up request in defined response contexts	Retrying immediately
`Cache-Control`	Cache directives	Blaming the origin before checking cached behavior
`Age`	How long a response has resided in a cache, when supplied	Assuming every cache emits it in every path
`Vary`	Request fields that affect cache selection	Comparing responses without matching relevant request headers
`Content-Type`	Declared media type	Parsing HTML as JSON because status was `200`
`Content-Encoding`	Representation coding such as compression	Comparing encoded bytes directly with decoded body size
`Set-Cookie`	Server-issued cookie state	Publishing it in logs or screenshots

`Via` and `Forwarded` are different signals

RFC 9110 defines Via for information about intermediate protocols and recipients along the request/response chain. It can help with forwarding diagnostics and loop detection.

Reference: RFC 9110 §7.6.3 — Via↗

RFC 7239 separately standardizes the optional Forwarded request header. It can disclose information changed or lost when a proxy participates in the request path, such as source-facing information or protocol details. The RFC also calls out privacy and security implications.

Reference: RFC 7239 — Forwarded HTTP Extension↗

Neither field is guaranteed to appear. A forward proxy, reverse proxy, CDN, load balancer, or application gateway can have its own forwarding behavior. Therefore:

plain text

no Via header != no intermediary
no Forwarded header != direct connection

Treat these fields as evidence when present, not proof when absent.

Redirect diagnostics

A surprising page often begins with a redirect chain.

bash

curl --silent --show-error \
  --location \
  --max-redirs 5 \
  --dump-header - \
  --output /dev/null \
  https://example.com/old-path

For a production investigation, preserve each hop rather than recording only the final status. Useful fields are:

status code
Location
hostname
response time
whether the request changed scheme or host
final URL

A 301/308 canonical migration is a different condition from an unexpected authentication redirect or a locale redirect.

Rate-limit diagnostics

RFC 6585 defines 429 Too Many Requests. The response may include Retry-After, but the RFC deliberately does not require a server to identify the client by IP alone. It can use other scopes such as credentials or cookies.

Reference: RFC 6585 §4 — 429 Too Many Requests↗

Capture:

plain text

status=429
retry_after=<value if present>
route_id=<redacted internal route identifier>
session_id=<hashed or non-secret identifier>
target_host=<host only>

Do not immediately change proxy IP and retry at full speed. First classify the rate limit, honor server instructions where applicable, and reduce load.

Proxy authentication: distinguish `407` from origin authentication

407 Proxy Authentication Required concerns proxy authentication. It is not the same as an origin site returning 401 Unauthorized.

When you see 407, check:

proxy hostname and port
username/password
percent-encoding or shell quoting
whether the client is actually using the expected proxy
whether the credential format carries country/session parameters
whether an upstream proxy changed the request path

Do not rotate IPs to solve an invalid proxy password.

Cache diagnostics

When a page looks stale, compare more than the body.

Capture:

bash

curl --silent --show-error \
  --dump-header /tmp/h1 \
  --output /tmp/b1 \
  https://example.com/resource

sleep 2

curl --silent --show-error \
  --dump-header /tmp/h2 \
  --output /tmp/b2 \
  https://example.com/resource

diff -u /tmp/h1 /tmp/h2 || true

Check fields such as Cache-Control, Age, ETag, Last-Modified, and Vary where present. Also record CDN-specific headers separately rather than treating them as universal HTTP semantics.

A stale response can originate in a browser cache, CDN, reverse proxy, framework cache, or application data cache. Header evidence helps narrow the layer, but application-level cache configuration is still required for a final diagnosis.

Content validation matters after the headers

A valid status and plausible headers do not mean the body is useful.

For scraping or monitoring, validate at least:

final URL
content type
required body markers
known login/challenge markers
locale/currency when relevant
schema or extraction invariants

Example in Python with HTTPX:

python

from dataclasses import dataclass
import httpx


@dataclass(frozen=True)
class CheckResult:
    status: int
    final_url: str
    content_type: str | None
    retry_after: str | None
    usable: bool


def check(url: str) -> CheckResult:
    with httpx.Client(timeout=15.0, follow_redirects=True) as client:
        response = client.get(url)

    content_type = response.headers.get("content-type")
    text = response.text.lower()

    usable = (
        response.status_code == 200
        and content_type is not None
        and "text/html" in content_type
        and "access denied" not in text
    )

    return CheckResult(
        status=response.status_code,
        final_url=str(response.url),
        content_type=content_type,
        retry_after=response.headers.get("retry-after"),
        usable=usable,
    )

The timeout and body marker are examples. Production validation must use target-specific evidence.

Never send secrets to an unknown public header checker

Request headers can contain secrets that are as powerful as passwords:

Authorization
Proxy-Authorization
Cookie
signed URLs
API keys in custom headers
session identifiers

Response headers can also contain sensitive values, especially Set-Cookie and application-specific tokens.

If you are diagnosing authenticated traffic, prefer local commands or an internal tool with explicit retention and access controls. Redact secrets before attaching evidence to tickets.

A useful incident record

Instead of a screenshot of random headers, store a small structured record:

json

{
  "observed_at": "2026-08-13T04:25:00Z",
  "method": "GET",
  "target_host": "example.com",
  "status": 429,
  "final_url": "https://example.com/resource",
  "content_type": "text/html",
  "retry_after": "120",
  "proxy_class": "residential",
  "attempt": 1,
  "validation": "rate_limited"
}

Exclude credentials and full cookie values.

Diagnostic checklist

Reproduce with the same HTTP method as the failing workload.

Capture the redirect chain and final URL.

Separate origin status from proxy authentication status.

Inspect Retry-After before retrying a 429/503 path.

Check cache and content-negotiation fields when responses differ.

Treat Via/Forwarded as optional evidence.

Validate the body, not only the HTTP status.

Redact authorization, cookies, proxy credentials, and signed URLs.

Store timestamps and route/session IDs so two attempts can be compared.

HTTP Header Checker for Proxy Diagnostics: What to Inspect and Why

Key Takeaways

A header check is evidence, not a verdict

Start with the right curl command

HEAD is not the same test as GET

A repeatable proxy-header test

Headers worth checking

`Via` and `Forwarded` are different signals

Redirect diagnostics

Rate-limit diagnostics

Proxy authentication: distinguish `407` from origin authentication

Cache diagnostics

Content validation matters after the headers

Never send secrets to an unknown public header checker

A useful incident record

Diagnostic checklist

Related BytesFlows guides

Alex Vance

Expand Your Knowledge

Proxy DNS Troubleshooting Checklist: NXDOMAIN, SOCKS5h, and IPv6

Proxy Environment Variables with curl: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY

Proxy TLS Certificate Errors: Diagnose Proxy vs Target TLS

Residential proxies for real data collection

Expand Your Knowledge

Proxy DNS Troubleshooting Checklist: NXDOMAIN, SOCKS5h, and IPv6

Proxy Environment Variables with curl: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY

Proxy TLS Certificate Errors: Diagnose Proxy vs Target TLS

Key Takeaways

A header check is evidence, not a verdict

Start with the right curl command

HEAD is not the same test as GET

A repeatable proxy-header test

Headers worth checking

Via and Forwarded are different signals

Redirect diagnostics

Rate-limit diagnostics

Proxy authentication: distinguish 407 from origin authentication

Cache diagnostics

Content validation matters after the headers

Never send secrets to an unknown public header checker

A useful incident record

Diagnostic checklist

Related BytesFlows guides

Alex Vance

Expand Your Knowledge

Proxy DNS Troubleshooting Checklist: NXDOMAIN, SOCKS5h, and IPv6

Proxy Environment Variables with curl: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY

Proxy TLS Certificate Errors: Diagnose Proxy vs Target TLS

Residential proxies for real data collection

Expand Your Knowledge

Proxy DNS Troubleshooting Checklist: NXDOMAIN, SOCKS5h, and IPv6

Proxy Environment Variables with curl: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY

Proxy TLS Certificate Errors: Diagnose Proxy vs Target TLS

`Via` and `Forwarded` are different signals

Proxy authentication: distinguish `407` from origin authentication