Proxy-Aware Idempotency Keys for Resilient Scraping

Why Idempotency Keys Matter When Proxies Rotate

Idempotency is often treated as a database concern, but proxy rotation makes it a distributed systems concern. When a scraper uses rotating proxies, each attempt can come from a different IP address. Network level retries may therefore happen after the first request already reached the server. Without an idempotency key, the server may process the same logical action twice.

The Hidden Duplicate Path

A single user action can trigger multiple HTTP calls. For example, a price tracking job may update a product row, send a webhook, and notify a warehouse. If the first update succeeds and the response is lost, a proxy retry can repeat it. Proxy rotation changes the source IP, but it does not change the application level operation. That means the retry policy must not create a new logical operation ID after a connection error. Instead, the same key should travel with the retry.

Key Design Rules

• Use a stable key per logical operation.
• Include operation type and entity ID.
• Add a deterministic suffix for retries.
• Use UUIDv7 for new operations.
• Store the key before sending when possible.

Backend Contract

The backend should return the original response for a repeated key. It should not create a second order, charge a card twice, or update inventory twice.

• Check key before processing.
• Return 200 or 409 for duplicates.
• Store the response body and status.
• Expire keys according to business needs.

Client Example

import uuid

def idempotency_key(operation, entity_id):
    return f'{operation}:{entity_id}:{uuid.uuid4()}'

For retries, use a deterministic key:

import uuid

def deterministic_key(operation, entity_id):
    return f'{operation}:{entity_id}:{uuid.uuid5(uuid.NAMESPACE_URL, entity_id)}'

Retry Logic With Proxy Rotation

import time
import requests

def post_with_retry(url, payload, key, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            response = requests.post(
                url,
                json=payload,
                headers={'Idempotency-Key': key},
                timeout=30,
            )
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException:
            if attempt == max_attempts - 1:
                raise
            time.sleep(2 ** attempt + 0.1)

When a proxy connection fails, reuse the same key. Do not generate a new key for the same logical operation. If the operation is genuinely new, create a new key before sending.

Why Proxy Rotation Intensifies Duplicate Risk

Proxy rotation is designed for scale. It spreads requests across IP pools. It also increases the chance that a retry happens after a timeout. Many scrapers treat a timeout as a retry. That is correct at the network layer, but not at the business layer. The same timeout can cause duplicate writes if the operation is not idempotent.

Multi Step Workflow Design

For multi step scraping workflows, scope keys carefully.

• Use a root key for the logical operation.
• Use child keys for side effects.
• Store parent key in job state.
• Pass parent key to downstream calls.

Example:

import uuid

def workflow_key(job_id, step_name):
    return f'{job_id}:{step_name}:{uuid.uuid5(uuid.NAMESPACE_URL, job_id + step_name)}'

Building An Idempotency Layer

A proxy-aware idempotency layer sits between the scraper and the target API. It assigns keys, stores receipts, and retries safely.

Key Storage Options

• Database table with unique key.
• Redis cache for fast lookup.
• Object storage for large receipts.

Retry Receipt Flow

1. Generate key.
2. Check receipt.
3. If absent, send request.
4. Store receipt.
5. Retry only if no receipt exists.

Proxy Pool Interaction

Rotating proxies may return connection errors. The idempotency layer should keep the key stable. The proxy pool should not know about the key. That keeps proxy concerns separate from business concerns.

Operational Checks

1. Send a request with a key.
2. Force a proxy failure before the request reaches the server.
3. Retry with the same key.
4. Assert that the backend processes the operation once.
5. Check proxy metadata does not bypass idempotency.

Logging And Metrics

Log the idempotency key with proxy metadata. This helps debugging. Do not let proxy pool metadata replace idempotency. Track duplicate key rates, retry counts, and operation outcomes.

Common Mistakes

• Generating a new key on every retry.
• Mixing operation types in one key namespace.
• Forgetting to store the key before sending.
• Assuming the backend already supports idempotency.
• Using proxy rotation as a substitute for idempotency.

Testing Duplicate Prevention

Use a local HTTP server or mock backend.

from http.server import BaseHTTPRequestHandler, HTTPServer
import json

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers['Content-Length'])
        body = self.rfile.read(length)
        key = self.headers.get('Idempotency-Key')
        # Store or return receipt
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(b'{}')

    def log_message(self, format, *args):
        pass

server = HTTPServer(('127.0.0.1', 8000), Handler)
server.serve_forever()

The handler is simplified. In production, store the key and response. Send the same key on retry.

Final Recommendation

Idempotency keys are a simple control. They make proxy retries safe. They make distributed scrapers easier to operate. Start with a clear operation model, add stable keys, and test retry behavior through rotating proxies. To implement this in a scraper, wrap the HTTP client. The wrapper creates keys, attaches headers, and delegates retries. The proxy client remains unaware of idempotency. This separation keeps the design clean. For high volume, use a database unique constraint on the key. This guarantees correctness even if the cache expires. Start with a small operation set, such as price updates and order creation, then expand. A proxy-aware idempotency layer should be tested with rotating pools. Use a proxy that fails after the request is sent. Verify the receipt store records one operation. This is the core value. Keep the key format versioned. Change the format only with a migration plan. Monitor key collisions. Keep retry budgets small. Log final state. It also supports HTTP, HTTPS, SOCKS5, and TCP routing because idempotency lives at the application layer.