Efficient Proxy Rotation in Go for High‑Throughput API Scraping

Introduction

When you’re pulling data from public or private APIs at scale, the bottleneck is rarely the API itself – it’s the IP limits and rate‑limiting mechanisms that throttle or block repeated requests. A thoughtful proxy‑rotation strategy lets you keep a steady flow of requests while staying under the radar of those limits.

In this post we walk through:

1. Why Go is a great fit for concurrent proxy rotation.
2. Designing a simple, yet robust, rotation engine.
3. Integrating error handling, back‑off, and persistence.
4. Real‑world patterns for >(10k req/min) performance.
5. A quick demo using RoProxy’s public API.

By the end you’ll have a reusable Go module you can drop into any scraping or monitoring project.

Why Go?

• Lightweight goroutines – thousands of lightweight threads (goroutines) can be spawned with minimal overhead.
• Excellent standard library – net/http supports custom transports, timeouts, and connection pooling out of the box.
• Static binaries – deployable as a single compiled artifact.
• Strong typing & concurrency primitives – channels, mutexes, and context make thread‑safe rotation trivial.

These traits combine to give you predictable latency and memory usage, which is essential when you’re trying to hit 10k+ requests per minute.

Core Concepts

Concept	What it solves	How it’s implemented in Go
Proxy pool	A pool of valid IP:port pairs	A slice of structs with metadata, protected by a sync.RWMutex
Round‑Robin / Weighted selection	Even distribution or priority handling	Simple index counter with modulo arithmetic, or a weighted slice
Failure tracking	Avoid repeatedly using a bad proxy	Map of proxy to failure count, reset after success
Back‑off	Reduce load on a failing API	time.Sleep with exponential back‑off, capped at a max
Persistence	Resume from last state on crash	File or KV store (e.g., BoltDB or Redis) that serialises the pool
Health‑check	Keep only live proxies	Periodic HEAD requests to a lightweight endpoint

Building the Rotation Engine

Below is a minimal yet extensible implementation. We’ll build a ProxyRotator type that exposes a Do(req) method. It hides all the rotation logic and lets the caller focus on request creation.

package main

import (
    "context"
    "crypto/tls"
    "fmt"
    "io"
    "net"
    "net/http"
    "sync"
    "time"
)

// Proxy represents a single HTTP proxy.
type Proxy struct {
    Address string // e.g., "http://1.2.3.4:8080"
    // You can add fields like Weight, LastUsed, Failures, etc.
}

// ProxyRotator manages a rotating pool.
type ProxyRotator struct {
    pool       []Proxy
    mu         sync.RWMutex
    idx        int
    httpClient *http.Client
}

// NewProxyRotator builds a client with a proxy pool.
func NewProxyRotator(proxies []Proxy) *ProxyRotator {
    tr := &http.Transport{}
    // Disable HTTP2 for simplicity; keep TCP keep‑alive
    tr.DisableCompression = true
    tr.DialContext = (&net.Dialer{Timeout: 10 * time.Second}).DialContext

    // Create a single client; transport will be swapped per request
    client := &http.Client{Transport: tr, Timeout: 30 * time.Second}

    return &ProxyRotator{pool: proxies, httpClient: client}
}

// rotate selects the next proxy in a thread‑safe way.
func (pr *ProxyRotator) rotate() Proxy {
    pr.mu.Lock()
    defer pr.mu.Unlock()
    proxy := pr.pool[pr.idx%len(pr.pool)]
    pr.idx++
    return proxy
}

// Do sends a request via a rotated proxy.
func (pr *ProxyRotator) Do(req *http.Request) (*http.Response, error) {
    proxy := pr.rotate()
    // Configure transport for this request
    transport := &http.Transport{
        Proxy: http.ProxyURL(proxyURL(proxy.Address)),
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
    }
    pr.httpClient.Transport = transport
    return pr.httpClient.Do(req)
}

func proxyURL(addr string) *url.URL {
    u, _ := url.Parse(addr)
    return u
}

Tip: The Transport is swapped per request to avoid state leaking between goroutines. In production you might pool transports or keep one per proxy.

Adding Resilience

The simple engine above will fail fast if a proxy is down. Below are a few tweaks.

Failure Tracking & Back‑off

// failureMap tracks consecutive failures.
var failureMap = struct{ mu sync.Mutex; m map[string]int }{m: make(map[string]int)}

func recordFailure(addr string) {
    failureMap.mu.Lock()
    defer failureMap.mu.Unlock()
    failureMap.m[addr]++
}

func recordSuccess(addr string) {
    failureMap.mu.Lock()
    defer failureMap.mu.Unlock()
    delete(failureMap.m, addr)
}

In the Do method, wrap the call with retry logic:

maxRetries := 3
var lastErr error
for i := 0; i < maxRetries; i++ {
    resp, err := pr.httpClient.Do(req)
    if err == nil {
        recordSuccess(proxy.Address)
        return resp, nil
    }
    lastErr = err
    recordFailure(proxy.Address)
    // Exponential back‑off
    time.Sleep(time.Duration(1<<i) * time.Second)
}
return nil, fmt.Errorf("all retries failed: %w", lastErr)

Health‑Check Routine

func (pr *ProxyRotator) StartHealthCheck(ctx context.Context) {
    go func() {
        ticker := time.NewTicker(5 * time.Minute)
        defer ticker.Stop()
        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
                pr.checkAll()
            }
        }
    }()
}

func (pr *ProxyRotator) checkAll() {
    for i, p := range pr.pool {
        if !pr.isAlive(p) {
            pr.mu.Lock()
            pr.pool = append(pr.pool[:i], pr.pool[i+1:]...)
            pr.mu.Unlock()
        }
    }
}

func (pr *ProxyRotator) isAlive(p Proxy) bool {
    testReq, _ := http.NewRequest("GET", "https://httpbin.org/ip", nil)
    transport := &http.Transport{Proxy: http.ProxyURL(proxyURL(p.Address))}
    client := &http.Client{Transport: transport, Timeout: 5 * time.Second}
    resp, err := client.Do(testReq)
    if err != nil { return false }
    resp.Body.Close()
    return resp.StatusCode == 200
}

Scaling to 10k+ Requests/Minute

1.  Goroutine Pool – Use a worker pool to limit concurrent requests and avoid exhausting system resources.

   workers := 200
   jobs := make(chan *http.Request, workers*2)
   var wg sync.WaitGroup
   for i := 0; i < workers; i++ {
       wg.Add(1)
       go func() {
           defer wg.Done()
           for req := range jobs {
               resp, err := rot.Do(req)
               // handle resp / err
           }
       }()
   }
   // enqueue jobs
   for _, url := range urls {
       jobs <- &http.Request{Method: "GET", URL: url}
   }
   close(jobs)
   wg.Wait()
   

2.  Connection Reuse – Reuse TLS sessions by sharing the same http.Transport for a given proxy. The trick is to store a map[proxy]Transport.

3.  Don’t Forget DNS – Use net.Resolver with PreferGo: true to avoid hitting system DNS cache, which can become stale under high churn.

4.  Monitorvaard – Export metrics (requests per proxy, failure rate, latency) to Prometheus. A simple counter per proxy gives you insights into uneven wear.

Using RoProxy in the Code

RoProxy offers a vetted pool of residential and datacenter proxies with a public API for health‑checks and rotation. Here’s how to pull a fresh pool and feed it into our ProxyRotator:

func fetchRoProxyPool() ([]Proxy, error) {
    // Example endpoint; replace with real RoProxy URL
    resp, err := http.Get("https://apiFlows.roProxy.com/proxies?limit=100")
    if err != nil { return nil, err }
    defer resp.Body.Close()
    var raw []struct{ IP string; Port string }
    if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { return nil, err }
    proxies := make([]Proxy, len(raw))
    for i, p := range raw {
        proxies[i] = Proxy{Address: fmt.Sprintf("http://%s:%s", p.IP, p.Port)}
    }
    return proxies, nil
}

Then initialize:

proxies, _ := fetchRoProxyPool()
rot := NewProxyRotator(proxies)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
rot.StartHealthCheck(ctx)

From here the rest of the code is the same. RoProxy’s health‑check endpoint ensures that you receive only live proxies, so your failure map stays small.

Common Pitfalls

Pitfall	Fix
Over‑rotating – rotating too fast causes the same IP to be reused before the API resets its counters.	Use a time‑based throttler: time.AfterFunc(time.Minute/requestsPerMinute, ...)
Ignoring TLS errors – many proxies use self‑signed certs.	Either trust the cert authority or skip verification (InsecureSkipVerify:true).
Blocking on a slow proxy – a single bad proxy can stall a goroutine.	Set per‑request timeouts and use a retry counter, as shown above.
Memory leak – unbounded http.Client pool.	Reuse transports per proxy or limit the number of concurrent clients.

Wrap‑Up

Efficient proxy rotation in Go boils down to a small set of patterns:

1. Thread‑safe pool – indexed round‑robin with a mutex.
2. Per‑request transport – swap the proxy per request.
3. ви Failure handling – retry with back‑off, track failures, and prune bad proxies.
4. Health checks – keep the pool lean and healthy.
5. Worker pool + metrics – scale without blowing the system.

When you combine these with a reliable provider like RoProxy, you can push past API rate limits, stay under the radar of anti‑scraping measures, and keep your data pipeline humming. Happy scraping!