Using Proxies for Canary Releases: Gradual API Rollout Across Geolocations

Why Use Proxies for Canary Releases

Canary releases let you expose a small fraction of users to a new version of an API or service while the majority continues to run the stable version. When your user base is global, you often want to test the new version in specific geographic markets before a worldwide launch. Proxies give you precise control over where the traffic appears to originate, letting you:

• Route a defined percentage of requests through proxies located in the target country or region.
• Mimic real user network characteristics (latency, ISP type) without provisioning actual servers there.
• Isolate the canary traffic from your internal monitoring systems, making it easier to measure impact.
• Quickly roll back by simply adjusting the proxy routing rules.

This guide walks through a practical, code‑first approach to implementing a proxy‑backed canary release for a REST API. We’ll cover proxy selection, traffic splitting, observability, and rollback procedures, with ready‑to‑run examples in Python and Node.js.

1. Architectural Overview

![Canary with proxies diagram]

1. API Gateway – Your entry point (e.g., Kong, AWS API Gateway, or a simple reverse proxy).
2. Traffic Splitter – A lightweight service that decides, per request, whether to send traffic to the stable backend, the canary backend, or to skip the split (100% stable).
3. Proxy Pool – A collection of residential or datacenter proxies, each tagged with its geographic location (country, city, ISP).
4. Canary Backend – The new version of your API, deployed alongside the stable version.
5. Observability – Metrics, logs, and traces collected separately for stable and canary streams.

The traffic splitter consults a configuration that defines:

• Canary percentage (e.g., 5% of total requests).
• Geographic filter (e.g., only requests that appear to come from Germany).
• Proxy selection strategy (round‑robin, latency‑based, etc.).

When a request matches the filter and is selected for the canary path, the splitter rewrites the X-Forwarded-For header (or uses the proxy’s Proxy-Authorization header) so the backend sees the request as originating from the chosen proxy.

2. Selecting the Right Proxy Type

Proxy Type	Best For	Pros	Cons
Residential IP (ISP‑assigned)	Mimicking real end‑users, bypassing geo‑blocks	High trust score, accurate geo‑data	Higher cost, limited pool size
Datacenter IP	High volume, low‑cost testing	Cheap, large pools, fast	Easily flagged by anti‑bot systems
Mobile IP (3G/4G/5G)	Testing carrier‑specific behavior	Real mobile NAT, useful for telco APIs	Expensive, often limited to certain regions

For most API canary scenarios, a mix of residential and datacenter proxies works well: use residential for the primary geo‑targeted canary (to avoid being blocked by strict rate‑limit or fraud detection) and datacenter for supplemental volume when you need to ramp up traffic quickly.

3. Building the Traffic Splitter

We’ll implement a simple splitter as a middleware that can be plugged into any HTTP framework. The logic is:

1. Read request headers (e.g., CF-IPCountry from Cloudflare, or rely on the proxy’s X-Forwarded-For).
2. If the request matches the geo filter and a random roll < canary_percentage, mark it for canary.
3. If marked, pick a proxy from the pool that matches the target country.
4. Forward the request to the backend via the chosen proxy, preserving original headers and adding any required proxy auth.

3.1 Python Example (using httpx)

# splitter.py
import random
import httpx
from typing import List, Dict

class ProxyCanarySplitter:
    def __init__(
        self,
        proxy_pool: List[Dict[str, str]],
        canary_percentage: float = 0.05,
        target_countries: List[str] = None,
    ):
        self.proxy_pool = proxy_pool
        self.canary_percentage = canary_percentage
        self.target_countries = set(target_countries or [])

    def _matches_geo(self, ip: str) -> bool:
        # In a real system you would call a geo‑IP service (e.g., MaxMind).
        # Here we assume the proxy dict already contains a 'country' field.
        # For demonstration we treat any request as matching if we have a proxy.
        return bool(self.target_countries)

    def _select_proxy(self, country: str) -> Dict[str, str]:
        candidates = [p for p in self.proxy_pool if p.get('country') == country]
        if not candidates:
            # fallback to any proxy
            candidates = self.proxy_pool
        return random.choice(candidates)

    async def forward(self, request: httpx.Request) -> httpx.Response:
        # 1. Decide if this request goes to canary
        if random.random() > self.canary_percentage:
            # stable path – send directly
            async with httpx.AsyncClient() as client:
                return await client.send(request)

        # 2. For simplicity we assume the request already came from a proxy
        #    that set X-Forwarded-For; we extract the country from that header.
        fwd = request.headers.get('x-forwarded-for', '').split(',')[0].strip()
        # In production you would resolve fwd → country via geo‑IP.
        # Here we just pick a random target country for demo.
        target_country = random.choice(list(self.target_countries)) if self.target_countries else 'us'

        proxy = self._select_proxy(target_country)
        proxy_url = f"http://{proxy['username']}:{proxy['password']}@{proxy['host']}:{proxy['port']}"

        async with httpx.AsyncClient(proxies={'http://': proxy_url, 'https://': proxy_url}) as client:
            # Preserve original headers; httpx will add Proxy‑Authorization automatically
            return await client.send(request)

# Usage example
if __name__ == '__main__':
    pool = [
        {'host': 'us-proxy.example.com', 'port': '3128', 'username': 'user1', 'password': 'pass1', 'country': 'us'},
        {'host': 'de-proxy.example.com', 'port': '3128', 'username': 'user2', 'password': 'pass2', 'country': 'de'},
    ]
    splitter = ProxyCanarySplitter(proxy_pool=pool, canary_percentage=0.1, target_countries=['de'])
    # Integrate splitter.forward() into your ASGI/WSGI middleware

3.2 Node.js Example (using axios and https-proxy-agent)

// splitter.js
const axios = require('axios');
const HttpsProxyAgent = require('https-proxy-agent');

function buildSplitter(proxyPool, options = {}) {
  const { canaryPercentage = 0.05, targetCountries = [] } = options;
  const targetSet = new Set(targetCountries.map(c => c.toLowerCase()));

  return async function split(requestConfig) {
    // 1. Decide canary vs stable
    if (Math.random() > canaryPercentage) {
      // stable – send directly
      return axios(requestConfig);
    }

    // 2. Determine target country from incoming request (simplified)
    const forwarded = requestConfig.headers?.['x-forwarded-for'];
    const ip = forwarded ? forwarded.split(',')[0].trim() : null;
    // In real code, look up ip → country via a geo‑IP service.
    // For demo we just pick the first target country.
    const targetCountry = targetCountries.length ? targetCountries[0] : 'us';

    // 3. Choose a proxy matching that country
    const candidates = proxyPool.filter(p => p.country.toLowerCase() === targetCountry.toLowerCase());
    const pool = candidates.length ? candidates : proxyPool;
    const proxy = pool[Math.floor(Math.random() * pool.length)];

    const proxyUrl = `http://${proxy.username}:${proxy.password}@${proxy.host}:${proxy.port}`;
    const agent = new HttpsProxyAgent(proxyUrl);

    // 4. Make the request via the proxy
    const proxyConfig = { ...requestConfig, httpAgent: agent, httpsAgent: agent };
    return axios(proxyConfig);
  };
}

// Example pool
const proxyPool = [
  { host: 'us-proxy.example.com', port: '3128', username: 'u1', password: 'p1', country: 'US' },
  { host: 'de-proxy.example.com', port: '3128', username: 'u2', password: 'p2', country: 'DE' },
];

const splitter = buildSplitter(proxyPool, { canaryPercentage: 0.1, targetCountries: ['DE'] });
// Use splitter({ method: 'GET', url: 'https://api.example.com/v1/resource', headers: { ... } })

4. Wiring the Splitter into Your API Gateway

If you already use an API gateway (Kong, Envoy, AWS API Gateway), you can implement the splitter as a custom plugin or Lambda@Edge function. The principle remains the same: evaluate the rule, pick a proxy, and call the upstream service via proxy directive.

4.1 Kong Plugin Skeleton (Lua)

-- canary_proxy.lua
local BasePlugin = require "kong.plugins.base_plugin"
local http = require "resty.http"
local cjson = require "cjson"

local CanaryProxyHandler = BasePlugin:extend()

CanaryProxyHandler.VERSION  = "1.0.0"
CanaryProxyHandler.PRIORITY = 1000

function CanaryProxyHandler:new()
  CanaryProxyHandler.super.new(self, "canary-proxy")
  self.proxy_pool = {
    { host = "de-proxy.example.com", port = 3128, username = "user2", password = "pass2", country = "DE" },
    -- add more...
  }
  self.canary_percentage = 0.1
  self.target_countries = { "DE" }
end

function CanaryProxyHandler:access(conf)
  CanaryProxyHandler.super.access(self)

  if math.random() > self.canary_percentage then
    -- stable path – let Kong continue upstream
    return
  end

  -- geo‑match (simplified: assume X-Forwarded-For already set)
  local fwd = ngx.req.get_headers()["x-forwarded-for"]
  if not fwd then return end
  local ip = fwd:match("^[^,]+")
  -- In production, lookup ip → country via a geo‑IP library.
  -- For demo we assume it matches.

  -- select proxy
  local candidates = {}
  for _, p in ipairs(self.proxy_pool) do
    if p.country == "DE" then table.insert(candidates, p) end
  end
  if #candidates == 0 then candidates = self.proxy_pool end
  local proxy = candidates[math.random(#candidates)]

  -- set proxy upstream
  ngx.set_proxy_uri("http://" .. proxy.host .. ":