[{"data":1,"prerenderedAt":24},["ShallowReactive",2],{"blog:post:en:proxy-aware-retry-queue-python":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":15,"thumbnail_url":20,"translations":21,"body":22,"asset_base":23},"proxy-aware-retry-queue-python","en","Building a Proxy-Aware Retry Queue in Python for Resilient Scraping","Learn how to implement a proxy-aware retry queue in Python that handles 429 and 503 responses with exponential backoff, jitter, and automatic proxy rotation for resilient web scraping.","2026-09-23",[10,11,12,13,14],"python","web scraping","proxy rotation","retry logic","rate limits",[10,16,17,18,19],"web-scraping","proxy-rotation","retry-logic","rate-limits","https://blog-api.ro-proxy.com/api/blog/posts/proxy-aware-retry-queue-python/thumbnail.svg?lang=en",[5],"## Why a Proxy-Aware Retry Queue Matters\n\nWhen you scrape at scale, two failure modes dominate: transient HTTP errors (429 Too Many Requests, 503 Service Unavailable) and proxy-level failures (timeouts, connection resets, 407 auth errors). Most scrapers handle these poorly — they either retry blindly with the same proxy (guaranteeing another ban) or give up entirely.\n\nA proxy-aware retry queue solves this by combining three techniques:\n\n- **Exponential backoff with jitter** to avoid thundering-herd retries\n- **Automatic proxy rotation** on each retry so you never hammer the same IP twice\n- **Error classification** to distinguish retryable from permanent failures\n\nThe result is a scraper that self-heals through transient failures without burning your proxy budget on doomed requests.\n\n## Core Design Principles\n\n### 1. Separate Retryable from Permanent Errors\n\nNot every error deserves a retry. A 404 means the resource is gone — retrying with a new proxy wastes credits. A 503 likely means the server is temporarily overloaded and will recover.\n\n**Retryable status codes:** 429, 500, 502, 503, 504\n**Permanent status codes:** 400, 401, 403, 404, 410\n\nFor proxy errors, treat timeouts and connection resets as retryable, but 407 (bad credentials) as permanent — rotating won't fix a wrong password.\n\n### 2. Rotate Proxies on Every Retry\n\nThe whole point of using proxies is distributing requests across IPs. If you retry with the same proxy that just got rate-limited, you're reinforcing the ban. Each retry attempt should pull a fresh proxy from your pool.\n\n### 3. Use Decorrelated Jitter\n\nPlain exponential backoff (1s, 2s, 4s, 8s...) causes synchronized retry storms when many requests fail simultaneously. Decorrelated jitter, popularized by AWS, adds randomness that spreads retries over time:\n\n```python\nimport random\n\ndef decorrelated_jitter(base: float, cap: float, previous_sleep: float) -> float:\n    \"\"\"AWS-style decorrelated jitter for backoff.\"\"\"\n    sleep = min(cap, random.uniform(base, previous_sleep * 3))\n    return sleep\n```\n\n## Implementation in Python\n\nHere's a complete, production-ready retry queue using `asyncio` and `aiohttp`:\n\n```python\nimport asyncio\nimport random\nfrom dataclasses import dataclass, field\nfrom typing import Optional\n\nimport aiohttp\n\n\nRETRYABLE_STATUS = {429, 500, 502, 503, 504}\nMAX_RETRIES = 5\nBASE_DELAY = 1.0\nMAX_DELAY = 60.0\n\n\n@dataclass\nclass ProxyPool:\n    \"\"\"Simple rotating proxy pool with round-robin selection.\"\"\"\n    proxies: list[str]\n    _index: int = 0\n    _lock: asyncio.Lock = field(default_factory=asyncio.Lock)\n\n    async def next(self) -> str:\n        async with self._lock:\n            proxy = self.proxies[self._index % len(self.proxies)]\n            self._index += 1\n            return proxy\n\n\n@dataclass\nclass RetryResult:\n    success: bool\n    status: Optional[int]\n    body: Optional[bytes]\n    attempts: int\n    error: Optional[str] = None\n\n\nasync def fetch_with_retry(\n    session: aiohttp.ClientSession,\n    url: str,\n    proxy_pool: ProxyPool,\n    method: str = \"GET\",\n    headers: Optional[dict] = None,\n    payload: Optional[dict] = None,\n) -> RetryResult:\n    \"\"\"\n    Fetch a URL with proxy rotation and exponential backoff.\n    \"\"\"\n    delay = BASE_DELAY\n    last_error = None\n\n    for attempt in range(1, MAX_RETRIES + 1):\n        proxy = await proxy_pool.next()\n        proxy_url = f\"http://{proxy}\"\n\n        try:\n            async with session.request(\n                method,\n                url,\n                proxy=proxy_url,\n                headers=headers,\n                json=payload,\n                timeout=aiohttp.ClientTimeout(total=30),\n            ) as response:\n                if response.status == 200:\n                    body = await response.read()\n                    return RetryResult(\n                        success=True,\n                        status=200,\n                        body=body,\n                        attempts=attempt,\n                    )\n\n                if response.status not in RETRYABLE_STATUS:\n                    # Permanent error — don't retry\n                    body = await response.read()\n                    return RetryResult(\n                        success=False,\n                        status=response.status,\n                        body=body,\n                        attempts=attempt,\n                        error=f\"HTTP {response.status} — not retryable\",\n                    )\n\n                # Retryable status code\n                last_error = f\"HTTP {response.status}\"\n\n        except asyncio.TimeoutError:\n            last_error = \"Timeout\"\n        except aiohttp.ClientError as e:\n            last_error = f\"ClientError: {e}\"\n\n        # Calculate next delay with decorrelated jitter\n        delay = min(MAX_DELAY, random.uniform(BASE_DELAY, delay * 3))\n\n        if attempt \u003C MAX_RETRIES:\n            await asyncio.sleep(delay)\n\n    return RetryResult(\n        success=False,\n        status=None,\n        body=None,\n        attempts=MAX_RETRIES,\n        error=last_error,\n    )\n```\n\n## Running the Queue Concurrently\n\nA single retry queue isn't enough — you need to process many URLs in parallel while respecting concurrency limits. Here's how to wire it up with a semaphore:\n\n```python\nasync def scrape_batch(\n    urls: list[str],\n    proxies: list[str],\n    concurrency: int = 20,\n) -> list[RetryResult]:\n    proxy_pool = ProxyPool(proxies)\n    semaphore = asyncio.Semaphore(concurrency)\n    connector = aiohttp.TCPConnector(limit=concurrency * 2, force_close=True)\n\n    async with aiohttp.ClientSession(connector=connector) as session:\n        async def bounded_fetch(url: str) -> RetryResult:\n            async with semaphore:\n                return await fetch_with_retry(session, url, proxy_pool)\n\n        results = await asyncio.gather(*[bounded_fetch(u) for u in urls])\n        return results\n\n\n# Example usage\nif __name__ == \"__main__\":\n    proxy_list = [\n        \"user:pass@proxy1.roproxy.com:8080\",\n        \"user:pass@proxy2.roproxy.com:8080\",\n        \"user:pass@proxy3.roproxy.com:8080\",\n    ]\n\n    target_urls = [\n        \"https://httpbin.org/status/200\",\n        \"https://httpbin.org/status/429\",\n        \"https://httpbin.org/status/503\",\n        \"https://httpbin.org/status/404\",\n    ]\n\n    results = asyncio.run(scrape_batch(target_urls, proxy_list))\n\n    for url, result in zip(target_urls, results):\n        status = \"OK\" if result.success else \"FAIL\"\n        print(f\"{status} | {url} | attempts={result.attempts} | error={result.error}\")\n```\n\n## Honoring Retry-After Headers\n\nMany APIs and CDNs return a `Retry-After` header with 429 and 503 responses. It tells you exactly how long to wait. Ignoring it and using your own backoff is wasteful — you'll either wait too long (wasting time) or too short (getting blocked again).\n\nUpdate the retry logic to respect this header:\n\n```python\nfrom email.utils import parsedate_to_datetime\nfrom datetime import datetime, timezone\n\n\ndef parse_retry_after(header_value: str) -> Optional[float]:\n    \"\"\"Parse Retry-After header (seconds or HTTP date).\"\"\"\n    if not header_value:\n        return None\n\n    # Numeric seconds\n    try:\n        return float(header_value)\n    except ValueError:\n        pass\n\n    # HTTP date format\n    try:\n        dt = parsedate_to_datetime(header_value)\n        if dt:\n            now = datetime.now(timezone.utc)\n            return max(0.0, (dt - now).total_seconds())\n    except (TypeError, ValueError):\n        pass\n\n    return None\n```\n\nThen inside `fetch_with_retry`, after detecting a retryable status:\n\n```python\nretry_after = parse_retry_after(response.headers.get(\"Retry-After\", \"\"))\nif retry_after is not None:\n    delay = min(MAX_DELAY, retry_after)\nelse:\n    delay = min(MAX_DELAY, random.uniform(BASE_DELAY, delay * 3))\n```\n\n## Tracking Proxy Health\n\nNot all proxies perform equally. Some are slow, some get banned faster. Track per-proxy success rates and deprioritize underperformers:\n\n```python\n@dataclass\nclass ProxyStats:\n    proxy: str\n    successes: int = 0\n    failures: int = 0\n    last_failure: float = 0.0\n\n    @property\n    def success_rate(self) -> float:\n        total = self.successes + self.failures\n        return self.successes / total if total > 0 else 1.0\n\n\nclass SmartProxyPool:\n    \"\"\"Proxy pool that tracks stats and deprioritizes bad proxies.\"\"\"\n\n    def __init__(self, proxies: list[str], min_success_rate: float = 0.5):\n        self._stats = {p: ProxyStats(proxy=p) for p in proxies}\n        self._min_success_rate = min_success_rate\n        self._lock = asyncio.Lock()\n\n    async def next(self) -> str:\n        async with self._lock:\n            # Sort by success rate, pick from top performers\n            sorted_proxies = sorted(\n                self._stats.values(),\n                key=lambda s: s.success_rate,\n                reverse=True,\n            )\n            # Filter out underperformers if enough alternatives exist\n            good = [s for s in sorted_proxies if s.success_rate >= self._min_success_rate]\n            pool = good if len(good) > len(self._stats) // 2 else sorted_proxies\n            return random.choice(pool).proxy\n\n    async def record_success(self, proxy: str):\n        async with self._lock:\n            self._stats[proxy].successes += 1\n\n    async def record_failure(self, proxy: str):\n        async with self._lock:\n            self._stats[proxy].failures += 1\n            self._stats[proxy].last_failure = asyncio.get_event_loop().time()\n```\n\nWire the stats recording into `fetch_with_retry` by calling `record_success` or `record_failure` after each attempt. This creates a feedback loop: proxies that consistently fail naturally fall out of rotation.\n\n## Common Pitfalls to Avoid\n\n### Don't Retry on 403\nA 403 often means the target detected your scraper pattern (headers, TLS fingerprint, request cadence) — not your IP. Rotating proxies won't help. Instead, review your headers and consider a headless browser approach.\n\n### Don't Retry Infinite Loops\nAlways cap retries. Five attempts is a reasonable maximum. Without a cap, a permanently broken endpoint will consume proxy credits forever.\n\n### Don't Share Sessions Across Proxies\nEach retry with a new proxy should use a fresh session or at least clear cookies. Carrying cookies from a banned IP to a new IP links the two and can get the new proxy banned too.\n\n### Don't Forget Force Close\nSet `force_close=True` on your `TCPConnector`. Some proxies don't handle keep-alive well, and reusing connections across rotated proxies causes bizarre routing errors.\n\n## Conclusion\n\nA proxy-aware retry queue is the difference between a scraper that survives transient failures and one that dies at the first 429. By combining error classification, decorrelated jitter, per-retry proxy rotation, and `Retry-After` header support, you get a system that's both resilient and respectful of target servers. Add per-proxy health tracking and you've got infrastructure that improves itself over time.\n","https://blog-api.ro-proxy.com/api/blog/posts/proxy-aware-retry-queue-python/assets",1790149642326]