Testing CDN Cache Invalidation Across Edge Nodes with Rotating Proxies

Why CDN Cache Invalidation Testing Matters

Content delivery networks cache assets at edge nodes around the world. When you deploy a new version of a CSS file, an API response, or a security header, you must be sure the purge reaches every PoP before users see stale content. Manual spot‑checks from a single location miss regional propagation delays, mis‑configured purge APIs, or edge‑logic bugs that only surface under real‑world routing.

Automating this verification gives you confidence that a cache‑clear operation is truly global, reduces the risk of serving outdated assets, and provides a measurable SLA for your CDN vendor.

Challenges of Global Verification

• Geographic diversity – You need requests from dozens of ISPs and continents.
• IP reputation – Datacenter IPs are often blocked or served a different cache layer than residential traffic.
• Session affinity – Some CDNs pin a client to a specific edge node for the life of a TCP connection; you must control whether you want to hit the same node repeatedly or spread across many.
• Header variability – Cache‑status headers (CF-Cache-Status, X-Cache, Fastly-Debug-Path, etc.) differ per vendor.
• Rate limits – Purging and verification APIs often enforce strict quotas.

Rotating residential proxies solve the first three problems: they give you a large pool of real‑user IPs, let you rotate per request, and optionally keep a sticky session when you need to probe a single edge node.

Choosing the Right Proxy Type

Proxy type	Best for	Drawbacks
Rotating residential	Broad geographic coverage, high trust score	Higher cost per GB
Static ISP (sticky)	Repeated hits to the same PoP for TTL measurement	Limited IP count
Mobile	Simulating mobile carrier routing (optional)	Very expensive, slower

For cache‑invalidation testing a rotating residential pool with an optional sticky‑session flag is the sweet spot.

Building a Rotating Proxy Pool

1. Select a provider that offers per‑request rotation and a sticky‑session endpoint (e.g., gate.smartproxy.com:7000 for rotation, gate.smartproxy.com:7001 for sticky).
2. Authenticate via username/password or IP‑whitelisting.
3. Wrap the proxy URL in a small helper that can switch between rotating and sticky modes.
4. Health‑check each proxy before use (HEAD request to a known CDN asset, expect 200 and a cache header).
5. Persist a list of healthy proxies in Redis or a local file for fast reuse across CI runs.

# proxy_pool.py
import requests
import random
import os

ROTATING = "http://user:pass@gate.smartproxy.com:7000"
STICKY   = "http://user:pass@gate.smartproxy.com:7001"

def get_proxy(sticky: bool = False) -> str:
    """Return a proxy URL. In sticky mode the same IP is kept for the session."""
    return STICKY if sticky else ROTATING

def healthy_proxies(url: str, sample: int = 20) -> list[str]:
    """Quick HEAD probe to filter dead IPs."""
    good = []
    for _ in range(sample):
        proxy = get_proxy()
        try:
            r = requests.head(url, proxies={"http": proxy, "https": proxy}, timeout=5)
            if r.status_code == 200 and any(h in r.headers for h in ("cf-cache-status", "x-cache", "fastly-debug-path")):
                good.append(proxy)
        except Exception:
            pass
    return good

Core Verification Logic

The verification steps are the same for every CDN, only the header names change.

Inspecting Cache‑Status Headers

• Cloudflare – CF-Cache-Status: HIT / MISS / EXPIRED / STALE
• Fastly – X-Cache: HIT / MISS and Fastly-Debug-Path
• Akamai – X-Cache: TCP_HIT / TCP_MISS

A successful purge should turn a HIT into a MISS (or EXPIRED) on the next request.

Measuring TTL and Stale‑While‑Revalidate

If you need to know how long an object stays cached after a purge, use a sticky session to hit the same edge node repeatedly and record the timestamp when the header flips back to HIT.

# verify_purge.py
import time
import requests
from proxy_pool import get_proxy, healthy_proxies

def check_cache_status(url: str, proxy: str, header_map: dict) -> str:
    """Return the cache status string for a given CDN."""
    r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=10)
    for hdr, key in header_map.items():
        if hdr in r.headers:
            return r.headers[hdr]
    return "UNKNOWN"

def wait_for_miss(url: str, proxy: str, header_map: dict, timeout: int = 120) -> bool:
    """Poll until the cache status becomes MISS/EXPIRED or timeout."""
    start = time.time()
    while time.time() - start < timeout:
        status = check_cache_status(url, proxy, header_map)
        if status.upper() in ("MISS", "EXPIRED", "STALE"):
            return True
        time.sleep(2)
    return False

if __name__ == "__main__":
    # Example for Cloudflare
    TEST_URL = "https://example.com/assets/main.css"
    HEADER_MAP = {"cf-cache-status": "CF-Cache-Status"}
    
    # 1. Warm the cache (first request should be MISS, second HIT)
    proxy = get_proxy(sticky=True)
    print("Warming cache...")
    requests.get(TEST_URL, proxies={"http": proxy, "https": proxy}, timeout=10)
    time.sleep(1)
    requests.get(TEST_URL, proxies={"http": proxy, "https": proxy}, timeout=10)
    
    # 2. Trigger purge via CDN API (omitted – use your provider's SDK)
    # purge_api.purge_url(TEST_URL)
    
    # 3. Verify propagation across many edge nodes
    proxies = healthy_proxies(TEST_URL, sample=30)
    results = {}
    for p in proxies:
        ok = wait_for_miss(TEST_URL, p, HEADER_MAP, timeout=60)
        results[p] = ok
        print(f"{p} -> {'PASS' if ok else 'FAIL'}")
    
    # 4. Summarize
    passed = sum(results.values())
    total = len(results)
    print(f"\nPropagation: {passed}/{total} edge nodes cleared cache.")
    if passed != total:
        exit(1)

Sticky Sessions for Single Edge Node

When you need to measure TTL or stale‑while‑revalidate windows, keep the same IP for the whole test:

proxy = get_proxy(sticky=True)   # same edge node for the duration
for i in range(10):
    status = check_cache_status(TEST_URL, proxy, HEADER_MAP)
    print(f"Attempt {i+1}: {status}")
    time.sleep(5)

Integrating Into CI/CD

1. Add a pipeline stage after your CDN purge step.
2. Store proxy credentials in the CI secret store (GitHub Actions secrets, GitLab CI variables, etc.).
3. Run the verification script as a container job; fail the build if any node returns FAIL.
4. Publish a JUnit‑style XML for test reporting (optional).

# .github/workflows/cdn-purge.yml
name: CDN Purge Verification
on:
  workflow_dispatch:
  push:
    branches: [main]
jobs:
  verify:
    runs-on: ubuntu-latest
    env:
      PROXY_USER: ${{ secrets.PROXY_USER }}
      PROXY_PASS: ${{ secrets.PROXY_PASS }}
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install deps
        run: pip install -r requirements.txt
      - name: Run verification
        run: python verify_purge.py

Real‑World Example: Cloudflare, Fastly, Akamai

CDN	Purge API	Cache‑status header	Typical propagation time
Cloudflare	POST /zones/:id/purge_cache	CF-Cache-Status	5‑30 s
Fastly	POST /service/:id/purge/:key	X-Cache, Fastly-Debug-Path	2‑15 s
Akamai	POST /ccu/v3/invalidate/url	X-Cache	10‑60 s

The same verify_purge.py works for all three – just swap HEADER_MAP and the purge trigger.

Best Practices and Gotchas

• Warm the cache first – a cold edge will always return MISS, giving a false positive.
• Use a sufficient sample size – 20‑30 proxies per region catches outliers.
• Respect provider rate limits – add jitter (time.sleep(random.uniform(0.5, 1.5))) between requests.
• Log raw headers for debugging; some CDNs add custom headers (CF-Ray, Fastly-Debug-Digest).
• Handle IPv6 – ensure your proxy pool supports IPv6 if the CDN serves over it.
• Don’t forget TLS verification – keep verify=True (default) to avoid man‑in‑the‑middle surprises.

Troubleshooting Checklist

1. All probes return UNKNOWN – proxy not reaching the CDN (check firewall, proxy auth).
2. Mixed PASS/FAIL – some PoPs haven’t received the purge yet; increase timeout or retry later.
3. Sticky session still hits different edges – provider may not honor sticky flag; confirm with vendor.
4. Header name mismatch – CDN may return a different header for non‑standard assets (e.g., X-Cache-Lookup). Adjust HEADER_MAP.
5. Rate‑limited by CDN – back‑off and reduce concurrency.

Conclusion

Automating CDN cache‑invalidation verification with rotating residential proxies turns a manual, error‑prone task into a reliable, repeatable gate in your deployment pipeline. By warming the cache, triggering a purge, and then probing dozens of real‑user IPs across the globe, you get concrete evidence that every edge node has flushed the stale object. The pattern—proxy pool → sticky vs. rotating mode → header inspection → CI integration—applies to any CDN and can be extended to test edge‑logic (WAF rules, A/B routing, signed URLs) with minimal code changes.

Start small: add a single verification job for your most critical asset, then expand the proxy sample and header matrix as confidence grows. Your users will never see a stale stylesheet again.