[{"data":1,"prerenderedAt":20},["ShallowReactive",2],{"blog:post:en:implementing-proxy-aware-token-bucket-rate-limiter-python-distributed-scrapers":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":15,"thumbnail_url":16,"translations":17,"body":18,"asset_base":19},"implementing-proxy-aware-token-bucket-rate-limiter-python-distributed-scrapers","en","Implementing a Proxy‑Aware Token Bucket Rate Limiter in Python for Distributed Scrapers","Learn how to build a token‑bucket rate limiter that respects per‑proxy limits, integrates with rotating proxy pools, and keeps high‑throughput scrapers stable.","2026-09-20",[10,11,12,13,14],"python","rate-limiting","proxy-rotation","token-bucket","scraping",[10,11,12,13,14],"https://blog-api.ro-proxy.com/api/blog/posts/implementing-proxy-aware-token-bucket-rate-limiter-python-distributed-scrapers/thumbnail.svg?lang=en",[5],"## Why a Token Bucket for Proxy‑Aware Scrapers\n\nWhen you run a distributed scraper behind a rotating proxy pool, each IP address has its own rate‑limit ceiling. A naïve `time.sleep()` between requests creates bursty traffic: a fast proxy may fire dozens of requests in a second while a slower one sits idle, quickly triggering bans. A token‑bucket algorithm solves this by granting each proxy a steady stream of \"tokens\" (request credits) that refill at a configurable rate. The scraper only sends a request when a token is available, guaranteeing smooth, predictable traffic per IP.\n\nKey benefits:\n- **Per‑proxy fairness** – every proxy obeys its own limit, preventing a single hot IP from exhausting the pool.\n- **Burst tolerance** – buckets can hold a small surplus, allowing short spikes (e.g., handling a redirect chain) without immediate throttling.\n- **Composability** – the limiter works with any async or sync HTTP client and plugs into existing proxy‑rotation logic.\n\n## Core Concepts\n\n### Token Bucket Algorithm\nA token bucket has three parameters:\n- `capacity` – maximum tokens the bucket can hold (burst allowance).\n- `refill_rate` – tokens added per second (steady‑state throughput).\n- `tokens` – current available tokens (float for sub‑second precision).\n\nOn each request the bucket attempts to `consume(1)`. If `tokens >= 1`, the request proceeds and `tokens -= 1`. Otherwise the caller waits until enough tokens accumulate (`wait_time = (1 - tokens) / refill_rate`).\n\n### Mapping Buckets to Proxies\nEach proxy in the pool gets its own bucket instance. When the rotation logic selects a proxy, the scraper also fetches the associated bucket. This design keeps the limiter stateless from the scraper’s perspective – it simply asks \"can I use this proxy now?\".\n\n## Implementation Overview\nWe’ll build four small, testable components:\n1. **`ProxyPool`** – yields `(proxy_url, proxy_id)` pairs and reports health.\n2. **`TokenBucket`** – core algorithm with async `take()` method.\n3. **`BucketManager`** – creates, stores, and retrieves buckets keyed by `proxy_id`.\n4. **`ScraperWorker`** – main loop that acquires a proxy, waits for its bucket, performs the request, and returns the proxy to the pool.\n\nAll code uses Python 3.11+, `asyncio`, and `aiohttp` for async HTTP. The same ideas translate to synchronous code or other languages.\n\n## Step‑by‑Step Implementation\n\n### 1. Proxy Pool Abstraction\n```python\n# proxy_pool.py\nimport random\nfrom typing import AsyncIterator, Tuple\n\nclass ProxyPool:\n    \"\"\"Simple round‑robin pool with health tracking.\"\"\"\n    def __init__(self, proxies: list[str]):\n        self._proxies = [{'url': p, 'id': i, 'healthy': True} for i, p in enumerate(proxies)]\n        self._index = 0\n\n    async def acquire(self) -> Tuple[str, int]:\n        \"\"\"Return next healthy proxy (url, id).\"\"\"\n        for _ in range(len(self._proxies)):\n            proxy = self._proxies[self._index]\n            self._index = (self._index + 1) % len(self._proxies)\n            if proxy['healthy']:\n                return proxy['url'], proxy['id']\n        raise RuntimeError('No healthy proxies available')\n\n    def mark_unhealthy(self, proxy_id: int):\n        for p in self._proxies:\n            if p['id'] == proxy_id:\n                p['healthy'] = False\n                break\n\n    def mark_healthy(self, proxy_id: int):\n        for p in self._proxies:\n            if p['id'] == proxy_id:\n                p['healthy'] = True\n                break\n```\n\n### 2. Token Bucket Class\n```python\n# token_bucket.py\nimport asyncio\nimport time\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass TokenBucket:\n    capacity: float          # max tokens\n    refill_rate: float       # tokens per second\n    _tokens: float = field(init=False)\n    _last_refill: float = field(init=False)\n    _lock: asyncio.Lock = field(default_factory=asyncio.Lock, init=False)\n\n    def __post_init__(self):\n        self._tokens = self.capacity\n        self._last_refill = time.monotonic()\n\n    def _refill(self):\n        now = time.monotonic()\n        elapsed = now - self._last_refill\n        new_tokens = elapsed * self.refill_rate\n        self._tokens = min(self.capacity, self._tokens + new_tokens)\n        self._last_refill = now\n\n    async def take(self, tokens: float = 1.0) -> None:\n        \"\"\"Block until `tokens` are available, then consume them.\"\"\"\n        async with self._lock:\n            while True:\n                self._refill()\n                if self._tokens >= tokens:\n                    self._tokens -= tokens\n                    return\n                # wait just enough for the missing tokens\n                deficit = tokens - self._tokens\n                wait = deficit / self.refill_rate\n                await asyncio.sleep(wait)\n```\n\n### 3. Bucket Manager\n```python\n# bucket_manager.py\nfrom token_bucket import TokenBucket\n\nclass BucketManager:\n    \"\"\"Holds a TokenBucket per proxy_id.\"\"\"\n    def __init__(self, capacity: float, refill_rate: float):\n        self._capacity = capacity\n        self._refill_rate = refill_rate\n        self._buckets: dict[int, TokenBucket] = {}\n\n    def get_bucket(self, proxy_id: int) -> TokenBucket:\n        if proxy_id not in self._buckets:\n            self._buckets[proxy_id] = TokenBucket(self._capacity, self._refill_rate)\n        return self._buckets[proxy_id]\n\n    def remove_bucket(self, proxy_id: int):\n        self._buckets.pop(proxy_id, None)\n```\n\n### 4. Scraper Worker Loop\n```python\n# scraper_worker.py\nimport asyncio\nimport aiohttp\nfrom proxy_pool import ProxyPool\nfrom bucket_manager import BucketManager\n\nclass ScraperWorker:\n    def __init__(\n        self,\n        pool: ProxyPool,\n        bucket_mgr: BucketManager,\n        target_urls: list[str],\n        concurrency: int = 10,\n    ):\n        self.pool = pool\n        self.bucket_mgr = bucket_mgr\n        self.target_urls = target_urls\n        self.semaphore = asyncio.Semaphore(concurrency)\n        self.session: aiohttp.ClientSession | None = None\n\n    async def _fetch(self, url: str, proxy_url: str, proxy_id: int) -> dict:\n        bucket = self.bucket_mgr.get_bucket(proxy_id)\n        await bucket.take()                     # rate‑limit per proxy\n        async with self.semaphore:\n            try:\n                async with self.session.get(url, proxy=proxy_url, timeout=10) as resp:\n                    data = await resp.text()\n                    return {'url': url, 'status': resp.status, 'len': len(data)}\n            except Exception as exc:\n                # on network error, mark proxy unhealthy and retry later\n                self.pool.mark_unhealthy(proxy_id)\n                raise\n\n    async def run(self):\n        self.session = aiohttp.ClientSession()\n        tasks = []\n        for url in self.target_urls:\n            tasks.append(asyncio.create_task(self._process_url(url)))\n        await asyncio.gather(*tasks, return_exceptions=True)\n        await self.session.close()\n\n    async def _process_url(self, url: str):\n        while True:\n            proxy_url, proxy_id = await self.pool.acquire()\n            try:\n                result = await self._fetch(url, proxy_url, proxy_id)\n                print(f\"[OK] {url} via proxy {proxy_id} -> {result['status']}\")\n                self.pool.mark_healthy(proxy_id)\n                break\n            except Exception:\n                # simple retry with next proxy\n                await asyncio.sleep(0.5)\n                continue\n```\n\n### 5. Wiring It Together\n```python\n# main.py\nimport asyncio\nfrom proxy_pool import ProxyPool\nfrom bucket_manager import BucketManager\nfrom scraper_worker import ScraperWorker\n\nPROXIES = [\n    \"http://user:pass@proxy1.example.com:8000\",\n    \"http://user:pass@proxy2.example.com:8000\",\n    \"http://user:pass@proxy3.example.com:8000\",\n]\nTARGETS = [f\"https://httpbin.org/get?page={i}\" for i in range(100)]\n\nasync def main():\n    pool = ProxyPool(PROXIES)\n    # 5 req/s per proxy, burst of 10\n    buckets = BucketManager(capacity=10, refill_rate=5)\n    worker = ScraperWorker(pool, buckets, TARGETS, concurrency=20)\n    await worker.run()\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## Handling Edge Cases\n\n| Situation | Strategy |\n|-----------|----------|\n| **Proxy becomes permanently banned** | After N consecutive failures, call `pool.mark_unhealthy(proxy_id)` and `bucket_mgr.remove_bucket(proxy_id)` to free memory. |\n| **Dynamic proxy addition** | `BucketManager.get_bucket` lazily creates a bucket, so new proxies work instantly. |\n| **Unequal proxy performance** | Tune `refill_rate` per‑proxy by storing a custom rate in `BucketManager` (e.g., a dict `proxy_id -> rate`). |\n| **Clock drift / async delays** | `time.monotonic()` is immune to system‑time changes; the lock inside `TokenBucket.take` serialises refills for a given proxy. |\n\n## Testing the Limiter\nA quick unit test verifies the bucket respects the configured rate.\n```python\n# test_token_bucket.py\nimport asyncio\nimport pytest\nfrom token_bucket import TokenBucket\n\n@pytest.mark.asyncio\nasync def test_refill_rate():\n    bucket = TokenBucket(capacity=5, refill_rate=10)  # 10 tokens/s\n    start = asyncio.get_event_loop().time()\n    for _ in range(5):\n        await bucket.take()          # should be instant (burst)\n    await bucket.take()              # 6th token -> wait ~0.1s\n    elapsed = asyncio.get_event_loop().time() - start\n    assert 0.09 \u003C elapsed \u003C 0.2      # allow scheduling jitter\n```\nRun with `pytest -q`.\n\n## Deploying in a Distributed Setting\nWhen workers run on multiple machines, each process would otherwise maintain its own bucket state, breaking the per‑IP contract. Two practical approaches:\n\n1. **Centralised Redis buckets** – store `tokens` and `last_refill` as a hash per `proxy_id`. Use a Lua script for atomic `take` (check‑and‑decrement). Workers call `EVALSHA` instead of local `TokenBucket.take()`.\n2. **Proxy‑side rate limiting** – many premium proxy providers (including RoProxy) enforce per‑IP limits upstream. In that case the local bucket acts only as a *client‑side* guard to avoid unnecessary 429 responses.\n\nExample Redis Lua snippet (atomic take):\n```lua\n-- KEYS[1] = proxy_id, ARGV[1] = capacity, ARGV[2] = refill_rate, ARGV[3] = now_ms\nlocal bucket = redis.call('HMGET', KEYS[1], 'tokens', 'last')\nlocal tokens = tonumber(bucket[1]) or tonumber(ARGV[1])\nlocal last = tonumber(bucket[2]) or tonumber(ARGV[3])\nlocal elapsed = (tonumber(ARGV[3]) - last) / 1000\ntokens = math.min(tonumber(ARGV[1]), tokens + elapsed * tonumber(ARGV[2]))\nif tokens >= 1 then\n    tokens = tokens - 1\n    redis.call('HMSET', KEYS[1], 'tokens', tokens, 'last', ARGV[3])\n    return 1\nelse\n    redis.call('HMSET', KEYS[1], 'tokens', tokens, 'last', ARGV[3])\n    return 0\nend\n```\nWorkers retry on `0` with a short `asyncio.sleep`.\n\n## Performance Tips\n- **Batch refill** – instead of refilling on every `take()`, schedule a background task that updates all buckets every 100 ms. Reduces lock contention.\n- **Use `asyncio.Queue` for URL distribution** – decouples URL production from consumption and naturally limits memory.\n- **Reuse `aiohttp.ClientSession`** – connection pooling across proxies (via `ProxyConnector`) cuts TLS handshake overhead.\n- **Monitor bucket health** – export `tokens` and `refill_rate` via Prometheus (`gauge` per proxy) to spot mis‑configured limits early.\n\n## Conclusion\nA token‑bucket rate limiter gives you deterministic, per‑proxy throughput control without the guesswork of fixed sleeps. By coupling a lightweight `TokenBucket` class with a `BucketManager` keyed to proxy identifiers, you can drop the component into any async scraper, scale it across machines with Redis, and keep your IP reputation intact. The pattern works equally well for SEO SERP monitoring, price‑intelligence pipelines, or ad‑verification bots – wherever you need high‑volume, polite traffic from a rotating proxy fleet.\n\nGive the code a spin, adjust `capacity` and `refill_rate` to match your provider’s limits, and watch the 429 errors disappear.\n","https://blog-api.ro-proxy.com/api/blog/posts/implementing-proxy-aware-token-bucket-rate-limiter-python-distributed-scrapers/assets",1790057930408]