Building a Proxy-Aware Retry Queue in Python for Resilient Scraping
23 September 2026
Why a Proxy-Aware Retry Queue Matters
When you scrape at scale, two failure modes dominate: transient HTTP errors (429 Too Many Requests, 503 Service Unavailable) and proxy-level failures (timeouts, connection resets, 407 auth errors). Most scrapers handle these poorly — they either retry blindly with the same proxy (guaranteeing another ban) or give up entirely.
A proxy-aware retry queue solves this by combining three techniques:
- Exponential backoff with jitter to avoid thundering-herd retries
- Automatic proxy rotation on each retry so you never hammer the same IP twice
- Error classification to distinguish retryable from permanent failures
The result is a scraper that self-heals through transient failures without burning your proxy budget on doomed requests.
Core Design Principles
1. Separate Retryable from Permanent Errors
Not every error deserves a retry. A 404 means the resource is gone — retrying with a new proxy wastes credits. A 503 likely means the server is temporarily overloaded and will recover.
Retryable status codes: 429, 500, 502, 503, 504 Permanent status codes: 400, 401, 403, 404, 410
For proxy errors, treat timeouts and connection resets as retryable, but 407 (bad credentials) as permanent — rotating won't fix a wrong password.
2. Rotate Proxies on Every Retry
The whole point of using proxies is distributing requests across IPs. If you retry with the same proxy that just got rate-limited, you're reinforcing the ban. Each retry attempt should pull a fresh proxy from your pool.
3. Use Decorrelated Jitter
Plain exponential backoff (1s, 2s, 4s, 8s...) causes synchronized retry storms when many requests fail simultaneously. Decorrelated jitter, popularized by AWS, adds randomness that spreads retries over time:
import random
def decorrelated_jitter(base: float, cap: float, previous_sleep: float) -> float:
"""AWS-style decorrelated jitter for backoff."""
sleep = min(cap, random.uniform(base, previous_sleep * 3))
return sleep
Implementation in Python
Here's a complete, production-ready retry queue using asyncio and aiohttp:
import asyncio
import random
from dataclasses import dataclass, field
from typing import Optional
import aiohttp
RETRYABLE_STATUS = {429, 500, 502, 503, 504}
MAX_RETRIES = 5
BASE_DELAY = 1.0
MAX_DELAY = 60.0
@dataclass
class ProxyPool:
"""Simple rotating proxy pool with round-robin selection."""
proxies: list[str]
_index: int = 0
_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
async def next(self) -> str:
async with self._lock:
proxy = self.proxies[self._index % len(self.proxies)]
self._index += 1
return proxy
@dataclass
class RetryResult:
success: bool
status: Optional[int]
body: Optional[bytes]
attempts: int
error: Optional[str] = None
async def fetch_with_retry(
session: aiohttp.ClientSession,
url: str,
proxy_pool: ProxyPool,
method: str = "GET",
headers: Optional[dict] = None,
payload: Optional[dict] = None,
) -> RetryResult:
"""
Fetch a URL with proxy rotation and exponential backoff.
"""
delay = BASE_DELAY
last_error = None
for attempt in range(1, MAX_RETRIES + 1):
proxy = await proxy_pool.next()
proxy_url = f"http://{proxy}"
try:
async with session.request(
method,
url,
proxy=proxy_url,
headers=headers,
json=payload,
timeout=aiohttp.ClientTimeout(total=30),
) as response:
if response.status == 200:
body = await response.read()
return RetryResult(
success=True,
status=200,
body=body,
attempts=attempt,
)
if response.status not in RETRYABLE_STATUS:
# Permanent error — don't retry
body = await response.read()
return RetryResult(
success=False,
status=response.status,
body=body,
attempts=attempt,
error=f"HTTP {response.status} — not retryable",
)
# Retryable status code
last_error = f"HTTP {response.status}"
except asyncio.TimeoutError:
last_error = "Timeout"
except aiohttp.ClientError as e:
last_error = f"ClientError: {e}"
# Calculate next delay with decorrelated jitter
delay = min(MAX_DELAY, random.uniform(BASE_DELAY, delay * 3))
if attempt < MAX_RETRIES:
await asyncio.sleep(delay)
return RetryResult(
success=False,
status=None,
body=None,
attempts=MAX_RETRIES,
error=last_error,
)
Running the Queue Concurrently
A single retry queue isn't enough — you need to process many URLs in parallel while respecting concurrency limits. Here's how to wire it up with a semaphore:
async def scrape_batch(
urls: list[str],
proxies: list[str],
concurrency: int = 20,
) -> list[RetryResult]:
proxy_pool = ProxyPool(proxies)
semaphore = asyncio.Semaphore(concurrency)
connector = aiohttp.TCPConnector(limit=concurrency * 2, force_close=True)
async with aiohttp.ClientSession(connector=connector) as session:
async def bounded_fetch(url: str) -> RetryResult:
async with semaphore:
return await fetch_with_retry(session, url, proxy_pool)
results = await asyncio.gather(*[bounded_fetch(u) for u in urls])
return results
# Example usage
if __name__ == "__main__":
proxy_list = [
"user:pass@proxy1.roproxy.com:8080",
"user:pass@proxy2.roproxy.com:8080",
"user:pass@proxy3.roproxy.com:8080",
]
target_urls = [
"https://httpbin.org/status/200",
"https://httpbin.org/status/429",
"https://httpbin.org/status/503",
"https://httpbin.org/status/404",
]
results = asyncio.run(scrape_batch(target_urls, proxy_list))
for url, result in zip(target_urls, results):
status = "OK" if result.success else "FAIL"
print(f"{status} | {url} | attempts={result.attempts} | error={result.error}")
Honoring Retry-After Headers
Many APIs and CDNs return a Retry-After header with 429 and 503 responses. It tells you exactly how long to wait. Ignoring it and using your own backoff is wasteful — you'll either wait too long (wasting time) or too short (getting blocked again).
Update the retry logic to respect this header:
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone
def parse_retry_after(header_value: str) -> Optional[float]:
"""Parse Retry-After header (seconds or HTTP date)."""
if not header_value:
return None
# Numeric seconds
try:
return float(header_value)
except ValueError:
pass
# HTTP date format
try:
dt = parsedate_to_datetime(header_value)
if dt:
now = datetime.now(timezone.utc)
return max(0.0, (dt - now).total_seconds())
except (TypeError, ValueError):
pass
return None
Then inside fetch_with_retry, after detecting a retryable status:
retry_after = parse_retry_after(response.headers.get("Retry-After", ""))
if retry_after is not None:
delay = min(MAX_DELAY, retry_after)
else:
delay = min(MAX_DELAY, random.uniform(BASE_DELAY, delay * 3))
Tracking Proxy Health
Not all proxies perform equally. Some are slow, some get banned faster. Track per-proxy success rates and deprioritize underperformers:
@dataclass
class ProxyStats:
proxy: str
successes: int = 0
failures: int = 0
last_failure: float = 0.0
@property
def success_rate(self) -> float:
total = self.successes + self.failures
return self.successes / total if total > 0 else 1.0
class SmartProxyPool:
"""Proxy pool that tracks stats and deprioritizes bad proxies."""
def __init__(self, proxies: list[str], min_success_rate: float = 0.5):
self._stats = {p: ProxyStats(proxy=p) for p in proxies}
self._min_success_rate = min_success_rate
self._lock = asyncio.Lock()
async def next(self) -> str:
async with self._lock:
# Sort by success rate, pick from top performers
sorted_proxies = sorted(
self._stats.values(),
key=lambda s: s.success_rate,
reverse=True,
)
# Filter out underperformers if enough alternatives exist
good = [s for s in sorted_proxies if s.success_rate >= self._min_success_rate]
pool = good if len(good) > len(self._stats) // 2 else sorted_proxies
return random.choice(pool).proxy
async def record_success(self, proxy: str):
async with self._lock:
self._stats[proxy].successes += 1
async def record_failure(self, proxy: str):
async with self._lock:
self._stats[proxy].failures += 1
self._stats[proxy].last_failure = asyncio.get_event_loop().time()
Wire the stats recording into fetch_with_retry by calling record_success or record_failure after each attempt. This creates a feedback loop: proxies that consistently fail naturally fall out of rotation.
Common Pitfalls to Avoid
Don't Retry on 403
A 403 often means the target detected your scraper pattern (headers, TLS fingerprint, request cadence) — not your IP. Rotating proxies won't help. Instead, review your headers and consider a headless browser approach.
Don't Retry Infinite Loops
Always cap retries. Five attempts is a reasonable maximum. Without a cap, a permanently broken endpoint will consume proxy credits forever.
Don't Share Sessions Across Proxies
Each retry with a new proxy should use a fresh session or at least clear cookies. Carrying cookies from a banned IP to a new IP links the two and can get the new proxy banned too.
Don't Forget Force Close
Set force_close=True on your TCPConnector. Some proxies don't handle keep-alive well, and reusing connections across rotated proxies causes bizarre routing errors.
Conclusion
A proxy-aware retry queue is the difference between a scraper that survives transient failures and one that dies at the first 429. By combining error classification, decorrelated jitter, per-retry proxy rotation, and Retry-After header support, you get a system that's both resilient and respectful of target servers. Add per-proxy health tracking and you've got infrastructure that improves itself over time.