Building a Proxy-Aware Request Deduplication Layer for Scrapers

Introduction

When a scraper runs at scale, the same URL can be queued many times — either because of retries, distributed workers, or overlapping crawl frontiers. Sending duplicate requests wastes bandwidth, burns proxy credits, and can trigger rate‑limit penalties. A request deduplication layer solves this by ensuring each logical request is executed once, while still cooperating with a rotating proxy service that may change the exit IP between attempts.

Why Request Deduplication Matters

• Cost control – Every outbound connection consumes a proxy session. Duplicate calls multiply that cost.
• Rate‑limit safety – Target sites often count requests per IP. Sending the same request from two different IPs looks like two distinct clients and can accelerate a block.
• Data consistency – Idempotent GET endpoints should return identical payloads. Re‑fetching adds no value but adds latency.
• Observability – A single canonical request per URL simplifies logging, metrics, and debugging.

Core Concepts

1. Canonical key – A deterministic hash of the request method, normalized URL, headers that affect the response (e.g., Accept-Language), and body (for POST).
2. In‑flight registry – A thread‑safe map from canonical key to a future or promise representing the ongoing request.
3. Result cache – Short‑lived storage (TTL 30‑60 s) of successful responses so that immediate retries reuse the cached payload.
4. Proxy session binding – When a request is first dispatched, bind it to a specific proxy session ID. If the session rotates (sticky → timed → per‑request), the binding follows the session’s current IP.

Architecture Overview

+----------------+      +---------------------+      +-------------------+
| Crawl Frontier | ---> | Deduplication Layer | ---> | Proxy Session Pool|
+----------------+      +---------------------+      +-------------------+
                               |  ^
                               |  |  (cache hit)
                               v  |
                        +-----------------+
                        | Response Cache  |
                        +-----------------+

• The Crawl Frontier pushes raw request objects.
• The Deduplication Layer computes the canonical key, checks the in‑flight registry and cache, and either returns a cached response, attaches to an in‑flight future, or creates a new outbound call.
• The Proxy Session Pool supplies a session identifier (e.g., SESSION_ID) and the corresponding gateway URL. The layer asks the pool for a session that matches the desired rotation mode.

Implementing the Deduplication Layer in Python

Below is a minimal, production‑ready implementation using only the standard library and requests. It demonstrates the registry, cache, and proxy binding.

import hashlib
import json
import threading
import time
from concurrent.futures import Future, ThreadPoolExecutor
from typing import Dict, Tuple, Any
import requests

# ----------------------------------------------------------------------
# Configuration placeholders – replace with real values at runtime
# ----------------------------------------------------------------------
GATEWAY_HOST = "GATEWAY_HOST"
HTTP_PORT = "HTTP_PORT"
SOCKS5_PORT = "SOCKS5_PORT"
USERNAME = "USERNAME"
PASSWORD = "PASSWORD"
API_KEY = "API_KEY"
SESSION_ID = "SESSION_ID"

PROXY_URL = f"http://{USERNAME}:{PASSWORD}@{GATEWAY_HOST}:{HTTP_PORT}"
PROXIES = {"http": PROXY_URL, "https": PROXY_URL}

# ----------------------------------------------------------------------
# Helper: canonical key
# ----------------------------------------------------------------------

def canonical_key(method: str, url: str, headers: dict, body: bytes = b"") -> str:
    """Return a stable SHA‑256 hash for the request."""
    # Normalise headers that affect variance
    relevant = {k.lower(): v for k, v in headers.items() if k.lower() in ("accept-language", "accept", "user-agent")}
    payload = json.dumps({
        "method": method.upper(),
        "url": url,
        "headers": relevant,
        "body": body.decode(errors="ignore")
    }, sort_keys=True).encode()
    return hashlib.sha256(payload).hexdigest()

# ----------------------------------------------------------------------
# Deduplication layer
# ----------------------------------------------------------------------

class RequestDeduplicator:
    def __init__(self, ttl: int = 45, max_workers: int = 10):
        self._inflight: Dict[str, Future] = {}
        self._cache: Dict[str, Tuple[float, Any]] = {}  # key -> (expires_at, response)
        self._lock = threading.Lock()
        self._ttl = ttl
        self._executor = ThreadPoolExecutor(max_workers=max_workers)

    def _make_request(self, method: str, url: str, headers: dict, body: bytes, session_id: str) -> requests.Response:
        """Execute a single HTTP call through the bound proxy session."""
        # In a real system you would fetch the current gateway for `session_id`
        # via the provider API. Here we reuse the static PROXIES dict.
        resp = requests.request(
            method=method,
            url=url,
            headers=headers,
            data=body if body else None,
            proxies=PROXIES,
            timeout=15,
        )
        resp.raise_for_status()
        return resp

    def fetch(self, method: str, url: str, headers: dict = None, body: bytes = b"",
              session_id: str = SESSION_ID) -> requests.Response:
        """Public API – returns a response, using deduplication and caching."""
        headers = headers or {}
        key = canonical_key(method, url, headers, body)
        now = time.time()

        # 1️⃣ Cache hit
        with self._lock:
            if key in self._cache:
                expires, cached = self._cache[key]
                if expires > now:
                    return cached
                else:
                    del self._cache[key]

        # 2️⃣ In‑flight attachment
        with self._lock:
            future = self._inflight.get(key)
            if future is not None:
                # Another thread is already fetching – wait for it
                pass
            else:
                future = self._executor.submit(self._make_request, method, url, headers, body, session_id)
                self._inflight[key] = future

        # 3️⃣ Wait for result
        try:
            resp = future.result()
        except Exception as exc:
            # On failure, remove from inflight so a retry can start fresh
            with self._lock:
                self._inflight.pop(key, None)
            raise

        # 4️⃣ Store in cache
        with self._lock:
            self._inflight.pop(key, None)
            self._cache[key] = (now + self._ttl, resp)
        return resp

# ----------------------------------------------------------------------
# Example usage
# ----------------------------------------------------------------------
if __name__ == "__main__":
    dedup = RequestDeduplicator()
    url = "https://api.ipify.org?format=json"
    headers = {"User-Agent": "dedup-demo/1.0", "Accept-Language": "en-US,en;q=0.9"}

    # First call – creates a request
    r1 = dedup.fetch("GET", url, headers=headers)
    print("First response:", r1.json())

    # Immediate second call – served from cache
    r2 = dedup.fetch("GET", url, headers=headers)
    print("Second response (cached):", r2.json())

Key implementation notes

• Thread safety – All shared structures are guarded by a single Lock. For higher throughput, replace with asyncio and asyncio.Lock.
• Cache TTL – 45 seconds works well for most scrape loops; adjust based on target freshness requirements.
• Proxy binding – The session_id argument lets the caller decide which rotating session to use. In a real deployment you would call the provider’s GET https://backend-api.ro-proxy.com/api/session to list sessions and pick one whose rotation mode matches the workload.
• Error handling – On any exception the in‑flight entry is removed, allowing a subsequent retry to launch a fresh request (potentially on a new IP after rotation).

Integrating with Rotating Proxy Sessions

When you configure a proxy, you receive a gateway host, HTTP port, and per‑session credentials. Build the proxy URL exactly as shown:

http://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT

If your plan includes a SOCKS5 port, you can also construct socks5://USERNAME:PASSWORD@GATEWAY_HOST:SOCKS5_PORT for libraries that support SOCKS5 (e.g., requests[socks]).

To rotate a session on demand (for example, after a 429 response), call the provider’s rotate endpoint:

import requests
rotate_url = f"https://backend-api.ro-proxy.com/api/session/{SESSION_ID}/rotate"
requests.post(rotate_url, headers={"Authorization": f"Bearer {API_KEY}"})

After rotation, the same SESSION_ID will now exit from a different IP while the deduplication layer continues to bind logical requests to that session identifier.

Handling Rotation Modes

Mode	Behavior	Deduplication impact
Sticky	IP stays until explicit rotate or failure	Cache remains valid for the whole sticky window.
Timed	IP changes automatically after set interval	Cache TTL should be ≤ rotation interval to avoid stale IPs.
Per‑request (Business)	New IP per connection	Bind each logical request to a fresh session or accept that the same session yields a new IP each call; cache only if the target tolerates IP variance.

Choose the mode that matches your idempotency guarantees. For strict idempotency (e.g., POST with side effects), prefer sticky sessions and disable caching for those keys.

Testing the Setup

1. Verify exit IP – After each fetch, call https://api.ipify.org?format=json through the same proxy to confirm the IP matches expectations.
2. Simulate rotation – Trigger the rotate endpoint and repeat the IP check; the deduplication layer should continue serving cached responses until TTL expires.
3. Load test – Fire concurrent duplicate URLs from multiple threads; ensure only one outbound request appears in the proxy logs.
4. Failure injection – Return 5xx from the target; confirm the layer retries on a new IP after rotation.

Conclusion

A lightweight deduplication layer eliminates wasteful duplicate traffic, respects rotating proxy session boundaries, and keeps scrape pipelines predictable. By canonicalising requests, tracking in‑flight futures, and caching with a short TTL, you gain cost savings and stability without sacrificing the flexibility of sticky, timed, or per‑request rotation. RoProxy provides the rotating sessions and programmatic rotation API that make this pattern straightforward to adopt in any Python‑based scraper.