Automating Cross-Region CSP and HSTS Checks with Proxies

Why CSP and HSTS Validation Needs a Global View

Content Security Policy (CSP) and HTTP Strict Transport Security (HSTS) are response headers that browsers enforce locally. A mis‑configured CSP can break legitimate scripts, while a missing or weak HSTS header leaves users vulnerable to downgrade attacks. Because CDNs, edge functions, and regional compliance rules often serve different header values, a single‑origin test is insufficient.

Challenges of Cross‑Region Header Inspection

• Geo‑based header variation – Edge nodes may add, strip, or modify CSP/HSTS based on local regulations.
• IP reputation – Some security services return stricter policies to known datacenter IPs.
• Rate limits & blocking – Repeated requests from one IP trigger WAF challenges.
• TLS negotiation differences – HSTS only applies over HTTPS; TLS version or cipher selection can affect header presence.

Rotating residential proxies solve these problems by providing clean, geographically diverse IPs that mimic real users.

Choosing the Right Proxy Type

Proxy type	Pros	Cons
Rotating residential	High trust score, real ISP ASNs, broad country coverage	Higher cost per GB
Static ISP	Consistent IP for session‑based checks	Limited geo diversity
Mobile	Best for mobile‑specific CSP rules	Expensive, smaller pools

For CSP/HSTS audits, rotating residential proxies give the best balance of trust and coverage.

Building a Minimal Validation Pipeline in Python

Below is a self‑contained script that:

1. Pulls a list of proxy endpoints (host:port:user:pass) from an environment variable.
2. Iterates over a target URL list.
3. Sends a HEAD request through each proxy.
4. Parses Content-Security-Policy and Strict-Transport-Security headers.
5. Emits JSON lines for downstream processing.

import os
import json
import random
import requests
from urllib.parse import urlparse

PROXY_LIST = os.getenv("ROPROXY_ENDPOINTS", "").split(",")
TARGETS = [
    "https://example.com",
    "https://shop.example.com",
    "https://api.example.com/health",
]

def build_proxy_dict(endpoint: str) -> dict:
    """Convert 'host:port:user:pass' into a requests‑compatible dict."""
    host, port, user, pwd = endpoint.split(":")
    proxy_url = f"http://{user}:{pwd}@{host}:{port}"
    return {"http": proxy_url, "https": proxy_url}

def fetch_headers(url: str, proxy: dict) -> dict:
    try:
        resp = requests.head(
            url,
            proxies=proxy,
            timeout=10,
            allow_redirects=True,
            headers={"User-Agent": "Mozilla/5.0 (compatible; CSP‑Auditor/1.0)"},
        )
        resp.raise_for_status()
        return {
            "url": url,
            "proxy": proxy["https"],
            "csp": resp.headers.get("Content-Security-Policy"),
            "hsts": resp.headers.get("Strict-Transport-Security"),
            "status": resp.status_code,
        }
    except Exception as exc:
        return {
            "url": url,
            "proxy": proxy["https"],
            "error": str(exc),
        }

def main():
    if not PROXY_LIST or PROXY_LIST == [""]:
        raise SystemExit("Set ROPROXY_ENDPOINTS with comma‑separated proxy strings.")

    for target in TARGETS:
        # pick a random proxy for each request to spread load
        proxy = build_proxy_dict(random.choice(PROXY_LIST).strip())
        result = fetch_headers(target, proxy)
        print(json.dumps(result))

if __name__ == "__main__":
    main()

Dissecting the CSP Header

A raw CSP header can contain multiple directives separated by semicolons. The helper below extracts each directive into a dictionary for easy policy comparison.

from collections import defaultdict

def parse_csp(header: str) -> dict:
    """Return {directive: [values]} mapping."""
    policies = defaultdict(list)
    for part in header.split(";"):
        part = part.strip()
        if not part:
            continue
        tokens = part.split()
        directive = tokens[0]
        values = tokens[1:]
        policies[directive].extend(values)
    return dict(policies)

# Example usage
csp_header = "default-src 'self'; script-src 'self' https://cdn.example.com; frame-ancestors 'none'"
print(parse_csp(csp_header))
# {'default-src': ["'self'"], 'script-src': ["'self'", 'https://cdn.example.com'], 'frame-ancestors': ["'none'"]}

Validating HSTS Directives

HSTS headers follow max-age=<seconds>; includeSubDomains; preload. The snippet below checks the three most common requirements.

def validate_hsts(header: str) -> dict:
    """Return a dict with boolean checks."""
    if not header:
        return {"present": False}
    parts = [p.strip() for p in header.split(";")]
    max_age = next((p for p in parts if p.startswith("max-age=")), None)
    include_sub = "includeSubDomains" in parts
    preload = "preload" in parts
    return {
        "present": True,
        "max_age_seconds": int(max_age.split("=")[1]) if max_age else None,
        "include_subdomains": include_sub,
        "preload": preload,
    }

Aggregating Results Across Regions

Run the script from multiple CI agents (GitHub Actions, GitLab CI, or a self‑hosted runner) each configured with a different proxy pool. Collect the JSON lines into a central store (e.g., Elasticsearch, BigQuery, or a simple SQLite DB) and query for anomalies:

SELECT url, proxy, csp, hsts
FROM audit_log
WHERE csp IS NULL OR hsts IS NULL
   OR json_extract(csp, '$.script-src') NOT LIKE '%cdn.example.com%';

Visualization tip: a Grafana dashboard with a world map panel colored by CSP compliance score gives stakeholders an instant health view.

CI/CD Integration Example (GitHub Actions)

name: CSP/HSTS Global Audit
on:
  schedule:
    - cron: '0 3 * * *'   # daily 03:00 UTC
  workflow_dispatch:

jobs:
  audit:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        region: [us-east, eu-west, ap-southeast]
    env:
      ROPROXY_ENDPOINTS: ${{ secrets[format('PROXY_{0}', matrix.region)] }}
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install deps
        run: pip install requests
      - name: Run auditor
        run: python auditor.py >> results-${{ matrix.region }}.jsonl
      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: audit-${{ matrix.region }}
          path: results-${{ matrix.region }}.jsonl

Each matrix job receives a region‑specific proxy list stored as a secret, guaranteeing geographic spread without code changes.

Best Practices & Gotchas

• Rotate per request, not per batch – prevents a single IP from being flagged.
• Respect Retry-After – some WAFs return 429 with a header; back off accordingly.
• Validate TLS – enable verify=True (default) and optionally pin the expected certificate SHA‑256 for high‑value targets.
• Handle redirects – allow_redirects=True follows edge‑location redirects; capture final URL to map policy to the correct hostname.
• Store raw headers – downstream diffing tools need the original string, not just parsed values.
• Throttle – even with residential IPs, keep QPS ≤ 5 per proxy to avoid ISP‑level rate limits.

Troubleshooting Checklist

1. Empty CSP/HSTS – confirm the target actually serves the headers over HTTPS; some sites only set them on specific paths.
2. Proxy authentication failures (407) – verify username/password encoding; special characters must be URL‑encoded.
3. TLS handshake errors – ensure the proxy supports the target’s TLS version (most residential pools support TLS 1.2+).
4. Inconsistent results across runs – enable sticky session for a single audit run (session_id parameter if your provider offers it) to isolate CDN caching effects.
5. Large response bodies – use HEAD to avoid downloading payloads; if HEAD is blocked, fall back to GET with stream=True and close immediately.

Extending the Framework

• Policy diffing – store the parsed CSP/HSTS per region and run a nightly diff to catch regressions.
• Alerting – integrate with PagerDuty or Slack when a region drops a critical directive (script-src 'self' missing).
• Automated remediation – feed failures into an IaC pipeline that updates edge‑function configs (e.g., Cloudflare Workers, AWS CloudFront Functions).
• Mobile‑specific checks – swap residential pool for a mobile proxy pool to validate CSP rules that differ for app‑webviews.

TL;DR

1. Deploy rotating residential proxies covering every region you serve.
2. Script HEAD requests through those proxies, capturing Content-Security-Policy and Strict-Transport-Security.
3. Parse and validate directives with the helper functions above.
4. Centralize JSON output, query for deviations, and visualize on a map.
5. Schedule the job in CI/CD, using per‑region proxy secrets for zero‑code geographic coverage.

By automating cross‑region CSP/HSTS audits you turn a manual, error‑prone checklist into a continuous security signal that scales with your global footprint.