[{"data":1,"prerenderedAt":20},["ShallowReactive",2],{"blog:post:en:proxy-aware-token-bucket-rate-limiter-go":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":15,"thumbnail_url":16,"translations":17,"body":18,"asset_base":19},"proxy-aware-token-bucket-rate-limiter-go","en","Proxy-Aware Token Bucket Rate Limiter in Go","Learn how to build a token bucket rate limiter in Go that integrates with rotating proxy sessions, automatically rotating IPs when limits are hit.","2026-11-01",[10,11,12,13,14],"go","rate-limiting","proxy","token-bucket","scraping",[10,11,12,13,14],"https://blog-api.ro-proxy.com/api/blog/posts/proxy-aware-token-bucket-rate-limiter-go/thumbnail.svg?lang=en",[5],"When you scrape or call APIs at scale, two constraints collide: the target’s rate limits and the proxy’s IP reputation. A token bucket gives you smooth, configurable throughput, while a rotating proxy lets you spread requests across many IPs. Combining them means you can back off locally before you hit a remote 429, and you can trigger a fresh IP the moment your bucket runs dry.\n\n## Understanding Token Bucket Algorithm\n\nThe token bucket is a classic rate‑limiting primitive. Imagine a bucket that holds *capacity* tokens. Every *refill interval* you add *rate* tokens (capped at capacity). Each request consumes one token. If the bucket is empty, the caller blocks until a token appears. This shape gives you burst tolerance (up to capacity) and a steady long‑term rate.\n\nKey parameters:\n- **capacity** – maximum burst size.\n- **rate** – tokens added per second.\n- **refill interval** – how often you add tokens (usually 1 s).\n\n## Designing a Proxy‑Aware Rate Limiter\n\nA proxy‑aware limiter has three responsibilities:\n1. **Enforce local rate limits** per logical client (e.g., per scraper worker).\n2. **Attach a proxy** to each outbound HTTP request.\n3. **Rotate the proxy IP** when the bucket is exhausted or when the remote server signals a block.\n\n### Core Components\n\n| Component | Role |\n|-----------|------|\n| `Limiter` | Holds a `rate.Limiter` per session and exposes `Wait(context)` |\n| `ProxyPool` | Stores proxy URLs and the associated session IDs |\n| `Rotator` | Calls the provider’s rotate endpoint when needed |\n\n### Integrating Proxy Rotation\n\nThe rotation trigger can be:\n- **Proactive**: after *N* requests or when `Wait` returns a long delay.\n- **Reactive**: on HTTP 429, 403, or connection errors.\n\nBoth strategies are easy to express in Go because the limiter’s `Wait` returns the time slept. If that time exceeds a threshold, you rotate.\n\n## Implementation in Go\n\nBelow is a complete, runnable example. It uses only the standard library and `golang.org/x/time/rate`.\n\n### Project Structure\n```\nproxy-limiter/\n├── main.go\n├── limiter.go\n├── proxy.go\n└── rotator.go\n```\n\n### Token Bucket Implementation\n**limiter.go**\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"time\"\n\t\"golang.org/x/time/rate\"\n)\n\n// SessionLimiter wraps a token bucket for a single proxy session.\ntype SessionLimiter struct {\n\tlimiter *rate.Limiter\n}\n\nfunc NewSessionLimiter(rps float64, burst int) *SessionLimiter {\n\treturn &SessionLimiter{limiter: rate.NewLimiter(rate.Limit(rps), burst)}\n}\n\n// Wait blocks until a token is available.\nfunc (sl *SessionLimiter) Wait(ctx context.Context) error {\n\treturn sl.limiter.Wait(ctx)\n}\n```\n\n### HTTP Client with Proxy\n**proxy.go**\n```go\npackage main\n\nimport (\n\t\"net/http\"\n\t\"net/url\"\n)\n\n// ProxyConfig holds the static parts of the proxy URL.\n// When using a rotating proxy service such as RoProxy, you configure the proxy URL as:\n// http://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT\n// Replace the placeholders with your actual credentials.\ntype ProxyConfig struct {\n\tGatewayHost string\n\tHTTPPort    string\n\tUsername    string\n\tPassword    string\n}\n\nfunc (pc *ProxyConfig) ProxyURL() *url.URL {\n\treturn &url.URL{\n\t\tScheme: \"http\",\n\t\tUser:   url.UserPassword(pc.Username, pc.Password),\n\t\tHost:   pc.GatewayHost + \":\" + pc.HTTPPort,\n\t}\n}\n\n// NewClient returns an http.Client that routes through the given proxy.\nfunc NewClient(proxyURL *url.URL) *http.Client {\n\ttransport := &http.Transport{\n\t\tProxy: http.ProxyURL(proxyURL),\n\t}\n\treturn &http.Client{Transport: transport, Timeout: 10 * time.Second}\n}\n```\n\n### Rotating IP via API\n**rotator.go**\n```go\npackage main\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"net/http\"\n\t\"time\"\n)\n\n// Rotator calls the provider’s rotate endpoint.\n// Only the two documented endpoints are used.\ntype Rotator struct {\n\tAPIKey     string\n\tSessionID  string\n\tHTTPClient *http.Client\n}\n\nfunc NewRotator(apiKey, sessionID string, client *http.Client) *Rotator {\n\treturn &Rotator{APIKey: apiKey, SessionID: sessionID, HTTPClient: client}\n}\n\n// Rotate triggers an immediate IP change for the session.\nfunc (r *Rotator) Rotate(ctx context.Context) error {\n\turl := \"https://backend-api.ro-proxy.com/api/session/\" + r.SessionID + \"/rotate\"\n\treq, err := http.NewRequestWithContext(ctx, \"POST\", url, nil)\n\tif err != nil {\n\t\treturn err\n\t}\n\treq.Header.Set(\"Authorization\", \"Bearer \"+r.APIKey)\n\treq.Header.Set(\"Content-Type\", \"application/json\")\n\n\tresp, err := r.HTTPClient.Do(req)\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode >= 300 {\n\t\treturn &rotateError{StatusCode: resp.StatusCode}\n\t}\n\treturn nil\n}\n\ntype rotateError struct { StatusCode int }\nfunc (e *rotateError) Error() string { return \"rotate failed: \" + string(rune(e.StatusCode)) }\n```\n\n### Putting It Together\n**main.go**\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"log\"\n\t\"net/http\"\n\t\"time\"\n)\n\nfunc main() {\n\t// --- Configuration (replace placeholders) ---\n\tproxyCfg := ProxyConfig{\n\t\tGatewayHost: \"GATEWAY_HOST\",\n\t\tHTTPPort:    \"HTTP_PORT\",\n\t\tUsername:    \"USERNAME\",\n\t\tPassword:    \"PASSWORD\",\n\t}\n\tapiKey := \"API_KEY\"\n\tsessionID := \"SESSION_ID\"\n\n\t// Build proxy‑aware HTTP client\n\tproxyURL := proxyCfg.ProxyURL()\n\tclient := NewClient(proxyURL)\n\n\t// Limiter: 5 requests/sec, burst of 10\n\tlimiter := NewSessionLimiter(5, 10)\n\n\t// Rotator uses a plain client (no proxy) to talk to the API\n\tapiClient := &http.Client{Timeout: 5 * time.Second}\n\trotator := NewRotator(apiKey, sessionID, apiClient)\n\n\t// Target endpoint for demo\n\ttarget := \"https://httpbin.org/ip\"\n\n\tfor i := 0; i \u003C 30; i++ {\n\t\tctx := context.Background()\n\t\t// Wait for token\n\t\tif err := limiter.Wait(ctx); err != nil {\n\t\t\tlog.Fatalf(\"limiter wait: %v\", err)\n\t\t}\n\n\t\t// Perform request\n\t\treq, _ := http.NewRequestWithContext(ctx, \"GET\", target, nil)\n\t\tresp, err := client.Do(req)\n\t\tif err != nil {\n\t\t\tlog.Printf(\"request error: %v\", err)\n\t\t\tcontinue\n\t\t}\n\t\t\n\t\t// Check for rate‑limit signals\n\t\tif resp.StatusCode == 429 || resp.StatusCode == 403 {\n\t\t\tlog.Printf(\"received %d, rotating IP\", resp.StatusCode)\n\t\t\tif err := rotator.Rotate(ctx); err != nil {\n\t\t\t\tlog.Printf(\"rotate failed: %v\", err)\n\t\t\t}\n\t\t\tresp.Body.Close()\n\t\t\tcontinue\n\t\t}\n\n\t\t// Read and log exit IP\n\t\tvar data map[string]string\n\t\tif err := json.NewDecoder(resp.Body).Decode(&data); err == nil {\n\t\t\tfmt.Printf(\"request %d: exit IP %s\\n\", i+1, data[\"origin\"])\n\t\t}\n\t\tresp.Body.Close()\n\n\t\t// Optional: proactive rotation after 15 requests\n\t\tif (i+1)%15 == 0 {\n\t\t\tlog.Printf(\"proactive rotation after %d requests\", i+1)\n\t\t\tif err := rotator.Rotate(ctx); err != nil {\n\t\t\t\tlog.Printf(\"rotate failed: %v\", err)\n\t\t\t}\n\t\t}\n\t}\n}\n```\n\n## Testing the Limiter\n\nRun the program and watch the output. You should see a steady stream of requests, each logging the exit IP returned by `httpbin.org/ip`. When a 429 or 403 appears, the rotator fires and the next request shows a new IP. The proactive rotation every 15 requests demonstrates how you can schedule IP changes based on your own heuristics.\n\n### Verify Exit IP\n\nThe example uses `https://httpbin.org/ip` (or `https://api.ipify.org?format=json`) to confirm the current proxy IP. Replace the target URL with your real endpoint when you go live.\n\n## Best Practices\n\n- **Separate limiters per session** – each proxy session gets its own bucket so a single noisy IP doesn’t starve the others.\n- **Tune burst vs. rate** – start with a low burst (5‑10) and increase only after you observe clean responses.\n- **Handle rotation errors** – if the rotate call fails, back off and retry; don’t hammer the API.\n- **Monitor latency** – long `Wait` times indicate you’re hitting the local limit; consider adding more sessions.\n- **Respect target policies** – even with rotation, honor `Retry-After` headers and robots.txt.\n\n## Conclusion\n\nA token bucket gives you predictable, configurable throughput, and coupling it with on‑demand IP rotation lets you stay under remote rate limits while keeping a healthy IP reputation. The pattern above works with any rotating proxy provider that offers a simple rotate endpoint. RoProxy provides rotating sessions that work well with this pattern, letting you focus on data collection instead of infrastructure plumbing.\n","https://blog-api.ro-proxy.com/api/blog/posts/proxy-aware-token-bucket-rate-limiter-go/assets",1793518611147]