[{"data":1,"prerenderedAt":20},["ShallowReactive",2],{"blog:post:en:building-proxy-aware-request-deduplication-layer":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":15,"thumbnail_url":16,"translations":17,"body":18,"asset_base":19},"building-proxy-aware-request-deduplication-layer","en","Building a Proxy-Aware Request Deduplication Layer for Scrapers","Learn how to design a deduplication layer that works with rotating proxy sessions, preventing duplicate work and handling sticky, timed, and per‑request IP rotation.","2026-10-28",[10,11,12,13,14],"proxy","deduplication","scraping","python","rotation",[10,11,12,13,14],"https://blog-api.ro-proxy.com/api/blog/posts/building-proxy-aware-request-deduplication-layer/thumbnail.svg?lang=en",[5],"## Introduction\n\nWhen a scraper runs at scale, the same URL can be queued many times — either because of retries, distributed workers, or overlapping crawl frontiers. Sending duplicate requests wastes bandwidth, burns proxy credits, and can trigger rate‑limit penalties. A request deduplication layer solves this by ensuring each logical request is executed once, while still cooperating with a rotating proxy service that may change the exit IP between attempts.\n\n## Why Request Deduplication Matters\n\n* **Cost control** – Every outbound connection consumes a proxy session. Duplicate calls multiply that cost.\n* **Rate‑limit safety** – Target sites often count requests per IP. Sending the same request from two different IPs looks like two distinct clients and can accelerate a block.\n* **Data consistency** – Idempotent GET endpoints should return identical payloads. Re‑fetching adds no value but adds latency.\n* **Observability** – A single canonical request per URL simplifies logging, metrics, and debugging.\n\n## Core Concepts\n\n1. **Canonical key** – A deterministic hash of the request method, normalized URL, headers that affect the response (e.g., `Accept-Language`), and body (for POST).\n2. **In‑flight registry** – A thread‑safe map from canonical key to a future or promise representing the ongoing request.\n3. **Result cache** – Short‑lived storage (TTL 30‑60 s) of successful responses so that immediate retries reuse the cached payload.\n4. **Proxy session binding** – When a request is first dispatched, bind it to a specific proxy session ID. If the session rotates (sticky → timed → per‑request), the binding follows the session’s current IP.\n\n## Architecture Overview\n\n```text\n+----------------+      +---------------------+      +-------------------+\n| Crawl Frontier | ---> | Deduplication Layer | ---> | Proxy Session Pool|\n+----------------+      +---------------------+      +-------------------+\n                               |  ^\n                               |  |  (cache hit)\n                               v  |\n                        +-----------------+\n                        | Response Cache  |\n                        +-----------------+\n```\n\n* The **Crawl Frontier** pushes raw request objects.\n* The **Deduplication Layer** computes the canonical key, checks the in‑flight registry and cache, and either returns a cached response, attaches to an in‑flight future, or creates a new outbound call.\n* The **Proxy Session Pool** supplies a session identifier (e.g., `SESSION_ID`) and the corresponding gateway URL. The layer asks the pool for a session that matches the desired rotation mode.\n\n## Implementing the Deduplication Layer in Python\n\nBelow is a minimal, production‑ready implementation using only the standard library and `requests`. It demonstrates the registry, cache, and proxy binding.\n\n```python\nimport hashlib\nimport json\nimport threading\nimport time\nfrom concurrent.futures import Future, ThreadPoolExecutor\nfrom typing import Dict, Tuple, Any\nimport requests\n\n# ----------------------------------------------------------------------\n# Configuration placeholders – replace with real values at runtime\n# ----------------------------------------------------------------------\nGATEWAY_HOST = \"GATEWAY_HOST\"\nHTTP_PORT = \"HTTP_PORT\"\nSOCKS5_PORT = \"SOCKS5_PORT\"\nUSERNAME = \"USERNAME\"\nPASSWORD = \"PASSWORD\"\nAPI_KEY = \"API_KEY\"\nSESSION_ID = \"SESSION_ID\"\n\nPROXY_URL = f\"http://{USERNAME}:{PASSWORD}@{GATEWAY_HOST}:{HTTP_PORT}\"\nPROXIES = {\"http\": PROXY_URL, \"https\": PROXY_URL}\n\n# ----------------------------------------------------------------------\n# Helper: canonical key\n# ----------------------------------------------------------------------\n\ndef canonical_key(method: str, url: str, headers: dict, body: bytes = b\"\") -> str:\n    \"\"\"Return a stable SHA‑256 hash for the request.\"\"\"\n    # Normalise headers that affect variance\n    relevant = {k.lower(): v for k, v in headers.items() if k.lower() in (\"accept-language\", \"accept\", \"user-agent\")}\n    payload = json.dumps({\n        \"method\": method.upper(),\n        \"url\": url,\n        \"headers\": relevant,\n        \"body\": body.decode(errors=\"ignore\")\n    }, sort_keys=True).encode()\n    return hashlib.sha256(payload).hexdigest()\n\n# ----------------------------------------------------------------------\n# Deduplication layer\n# ----------------------------------------------------------------------\n\nclass RequestDeduplicator:\n    def __init__(self, ttl: int = 45, max_workers: int = 10):\n        self._inflight: Dict[str, Future] = {}\n        self._cache: Dict[str, Tuple[float, Any]] = {}  # key -> (expires_at, response)\n        self._lock = threading.Lock()\n        self._ttl = ttl\n        self._executor = ThreadPoolExecutor(max_workers=max_workers)\n\n    def _make_request(self, method: str, url: str, headers: dict, body: bytes, session_id: str) -> requests.Response:\n        \"\"\"Execute a single HTTP call through the bound proxy session.\"\"\"\n        # In a real system you would fetch the current gateway for `session_id`\n        # via the provider API. Here we reuse the static PROXIES dict.\n        resp = requests.request(\n            method=method,\n            url=url,\n            headers=headers,\n            data=body if body else None,\n            proxies=PROXIES,\n            timeout=15,\n        )\n        resp.raise_for_status()\n        return resp\n\n    def fetch(self, method: str, url: str, headers: dict = None, body: bytes = b\"\",\n              session_id: str = SESSION_ID) -> requests.Response:\n        \"\"\"Public API – returns a response, using deduplication and caching.\"\"\"\n        headers = headers or {}\n        key = canonical_key(method, url, headers, body)\n        now = time.time()\n\n        # 1️⃣ Cache hit\n        with self._lock:\n            if key in self._cache:\n                expires, cached = self._cache[key]\n                if expires > now:\n                    return cached\n                else:\n                    del self._cache[key]\n\n        # 2️⃣ In‑flight attachment\n        with self._lock:\n            future = self._inflight.get(key)\n            if future is not None:\n                # Another thread is already fetching – wait for it\n                pass\n            else:\n                future = self._executor.submit(self._make_request, method, url, headers, body, session_id)\n                self._inflight[key] = future\n\n        # 3️⃣ Wait for result\n        try:\n            resp = future.result()\n        except Exception as exc:\n            # On failure, remove from inflight so a retry can start fresh\n            with self._lock:\n                self._inflight.pop(key, None)\n            raise\n\n        # 4️⃣ Store in cache\n        with self._lock:\n            self._inflight.pop(key, None)\n            self._cache[key] = (now + self._ttl, resp)\n        return resp\n\n# ----------------------------------------------------------------------\n# Example usage\n# ----------------------------------------------------------------------\nif __name__ == \"__main__\":\n    dedup = RequestDeduplicator()\n    url = \"https://api.ipify.org?format=json\"\n    headers = {\"User-Agent\": \"dedup-demo/1.0\", \"Accept-Language\": \"en-US,en;q=0.9\"}\n\n    # First call – creates a request\n    r1 = dedup.fetch(\"GET\", url, headers=headers)\n    print(\"First response:\", r1.json())\n\n    # Immediate second call – served from cache\n    r2 = dedup.fetch(\"GET\", url, headers=headers)\n    print(\"Second response (cached):\", r2.json())\n```\n\n### Key implementation notes\n\n* **Thread safety** – All shared structures are guarded by a single `Lock`. For higher throughput, replace with `asyncio` and `asyncio.Lock`.\n* **Cache TTL** – 45 seconds works well for most scrape loops; adjust based on target freshness requirements.\n* **Proxy binding** – The `session_id` argument lets the caller decide which rotating session to use. In a real deployment you would call the provider’s `GET https://backend-api.ro-proxy.com/api/session` to list sessions and pick one whose rotation mode matches the workload.\n* **Error handling** – On any exception the in‑flight entry is removed, allowing a subsequent retry to launch a fresh request (potentially on a new IP after rotation).\n\n## Integrating with Rotating Proxy Sessions\n\nWhen you configure a proxy, you receive a gateway host, HTTP port, and per‑session credentials. Build the proxy URL exactly as shown:\n\n```text\nhttp://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT\n```\n\nIf your plan includes a SOCKS5 port, you can also construct `socks5://USERNAME:PASSWORD@GATEWAY_HOST:SOCKS5_PORT` for libraries that support SOCKS5 (e.g., `requests[socks]`).\n\nTo rotate a session on demand (for example, after a 429 response), call the provider’s rotate endpoint:\n\n```python\nimport requests\nrotate_url = f\"https://backend-api.ro-proxy.com/api/session/{SESSION_ID}/rotate\"\nrequests.post(rotate_url, headers={\"Authorization\": f\"Bearer {API_KEY}\"})\n```\n\nAfter rotation, the same `SESSION_ID` will now exit from a different IP while the deduplication layer continues to bind logical requests to that session identifier.\n\n## Handling Rotation Modes\n\n| Mode      | Behavior                                    | Deduplication impact                                 |\n|-----------|---------------------------------------------|------------------------------------------------------|\n| Sticky    | IP stays until explicit rotate or failure   | Cache remains valid for the whole sticky window.     |\n| Timed     | IP changes automatically after set interval | Cache TTL should be ≤ rotation interval to avoid stale IPs. |\n| Per‑request (Business) | New IP per connection                | Bind each logical request to a fresh session or accept that the same session yields a new IP each call; cache only if the target tolerates IP variance. |\n\nChoose the mode that matches your idempotency guarantees. For strict idempotency (e.g., POST with side effects), prefer sticky sessions and disable caching for those keys.\n\n## Testing the Setup\n\n1. **Verify exit IP** – After each fetch, call `https://api.ipify.org?format=json` through the same proxy to confirm the IP matches expectations.\n2. **Simulate rotation** – Trigger the rotate endpoint and repeat the IP check; the deduplication layer should continue serving cached responses until TTL expires.\n3. **Load test** – Fire concurrent duplicate URLs from multiple threads; ensure only one outbound request appears in the proxy logs.\n4. **Failure injection** – Return 5xx from the target; confirm the layer retries on a new IP after rotation.\n\n## Conclusion\n\nA lightweight deduplication layer eliminates wasteful duplicate traffic, respects rotating proxy session boundaries, and keeps scrape pipelines predictable. By canonicalising requests, tracking in‑flight futures, and caching with a short TTL, you gain cost savings and stability without sacrificing the flexibility of sticky, timed, or per‑request rotation. RoProxy provides the rotating sessions and programmatic rotation API that make this pattern straightforward to adopt in any Python‑based scraper.\n","https://blog-api.ro-proxy.com/api/blog/posts/building-proxy-aware-request-deduplication-layer/assets",1793170431830]