[{"data":1,"prerenderedAt":19},["ShallowReactive",2],{"blog:post:en:using-rotating-proxy-python-urllib":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":14,"thumbnail_url":15,"translations":16,"body":17,"asset_base":18},"using-rotating-proxy-python-urllib","en","Using a Rotating Proxy with Python's urllib.request","Send HTTP and HTTPS traffic through a rotating proxy using only the Python standard library, with auth, IP checks, and rotation logic.","2026-11-02",[10,11,12,13],"python","proxy","urllib","rotation",[10,11,12,13],"https://blog-api.ro-proxy.com/api/blog/posts/using-rotating-proxy-python-urllib/thumbnail.svg?lang=en",[5],"Many scraping and automation scripts start with the `requests` library, but Python's standard library includes `urllib.request`, which works without installing anything. That matters in locked-down build images, AWS Lambda layers, or internal tools where adding dependencies is slow or blocked. A rotating proxy sits between your script and the target, swapping the exit IP on a schedule or per connection. This guide shows how to route `urllib.request` through a rotating proxy, add authentication, verify the exit IP, and rotate the IP between requests using only built-in modules.\n\n## Why Use urllib.request with a Rotating Proxy\n\n`urllib.request` is part of Python's standard library. It supports HTTP and HTTPS, proxy handlers, and basic authentication out of the box. When you pair it with a rotating proxy, each request can appear to come from a different IP address. This helps avoid rate limits, access geo-restricted content, and distribute load across multiple exit points.\n\nThe main trade-off is convenience. `urllib.request` is more verbose than `requests` or `httpx`. It does not automatically decode JSON, manage sessions, or retry on connection errors. You write that logic yourself. For a focused integration with a rotating proxy, that verbosity is actually useful: it makes exactly when and how the proxy is used explicit in your code.\n\n## How the Proxy Connection Works\n\nA forward proxy accepts a request from your client, connects to the destination on your behalf, and returns the response. For HTTPS, your client sends a `CONNECT` request through the proxy to establish a tunnel, then performs TLS directly with the target. The proxy sees the destination host but not the encrypted payload.\n\nYour proxy session provides a gateway host and port. In code, the proxy URL looks like this:\n\n```\nhttp://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT\n```\n\nUse the same URL for both `http` and `https` keys. The scheme inside the proxy URL remains `http` even when the final destination is HTTPS, because you are talking to the proxy over plain HTTP and asking it to open a tunnel.\n\nIf your plan includes a SOCKS5 port, `urllib.request` does not support SOCKS5 natively. You would need an external wrapper or a different library. This guide uses the HTTP port, which works for both HTTP and HTTPS targets.\n\nDo not mix proxied and direct requests through the same opener unless you intend to. If part of your script should call internal APIs directly, create a second opener without the `ProxyHandler` and use it for those calls.\n\n## Configure the Proxy Handler\n\nCreate a `ProxyHandler` and pass it to `build_opener`. The opener replaces the default URL opener so every request you make through it uses the proxy.\n\n```\nimport urllib.request\n\nproxy_url = 'http://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT'\n\nproxy_handler = urllib.request.ProxyHandler({\n    'http': proxy_url,\n    'https': proxy_url,\n})\n\nopener = urllib.request.build_opener(proxy_handler)\n```\n\nIf you use RoProxy, each session has its own username and password. Insert those credentials into `USERNAME` and `PASSWORD`. The gateway host and HTTP port come from your session details. Sessions on Pro and Business plans also expose a SOCKS5 port, but this example uses the HTTP port.\n\nInstall the opener globally if you want every `urllib.request.urlopen` call in the process to use the proxy:\n\n```\nurllib.request.install_opener(opener)\n```\n\nOtherwise, call `opener.open(request)` directly for scoped proxy use.\n\n### Add Basic Authentication\n\nThe username and password inside the proxy URL handle proxy authentication. If the proxy rejects them, you receive an HTTP 407 error. Do not hardcode credentials in version control. Read them from environment variables:\n\n```\nimport os\n\nusername = os.environ['PROXY_USERNAME']\npassword = os.environ['PROXY_PASSWORD']\ngateway_host = os.environ['PROXY_GATEWAY_HOST']\nhttp_port = os.environ['PROXY_HTTP_PORT']\n\nproxy_url = f'http://{username}:{password}@{gateway_host}:{http_port}'\n```\n\nIf the password contains special characters, percent-encode it with `urllib.parse.quote` before inserting it into the URL.\n\n### Test the Exit IP\n\nBefore rotating anything, confirm your traffic actually leaves through the proxy. Request an IP echo service and compare the result to your local address.\n\n```\nimport json\n\nrequest = urllib.request.Request('https://api.ipify.org?format=json')\n\nwith opener.open(request, timeout=10) as response:\n    data = json.loads(response.read().decode())\n    print('Exit IP:', data['ip'])\n```\n\nRun this twice with different proxy sessions. If the IP changes, the proxy is working. If it matches your local IP, the proxy settings were ignored or bypassed.\n\n## Rotate the IP Between Requests\n\nRotation behavior depends on the session mode:\n\n- **Sticky:** Keeps the same IP until it rotates, fails, or you trigger a change. Use this for multi-page flows where the IP must stay constant.\n- **Timed:** Changes the IP after an interval you set. Use this for long-running jobs that need a fresh address periodically.\n- **Per request:** A new IP for every new connection. Available on the Business plan only. Use this for stateless checks where each request should appear independent.\n\n`urllib.request` does not rotate IPs itself. Rotation happens at the proxy layer. Your code controls when to ask for a new IP by calling the rotation endpoint.\n\n```\nimport urllib.request\n\nrotate_url = 'https://backend-api.ro-proxy.com/api/session/SESSION_ID/rotate'\n\nrequest = urllib.request.Request(rotate_url, method='POST')\nrequest.add_header('Authorization', 'Bearer API_KEY')\n\nwith urllib.request.urlopen(request, timeout=10) as response:\n    print('Rotation status:', response.status)\n```\n\nReplace `SESSION_ID` with the identifier for the session you want to rotate. A successful response means the session now uses a different IP for new connections.\n\n## Handle Errors and Retries\n\nProxy connections fail for reasons outside your control: the target blocks the exit IP, the proxy gateway is unreachable, or authentication expires. Wrap proxy calls in try/except blocks and distinguish between proxy errors and target errors.\n\n```\nfrom urllib.error import URLError, HTTPError\n\ntry:\n    with opener.open(request, timeout=15) as response:\n        body = response.read().decode()\nexcept HTTPError as e:\n    print(f'Target returned {e.code}: {e.reason}')\nexcept URLError as e:\n    print(f'Proxy or network failure: {e.reason}')\n```\n\nAn HTTP 407 means proxy authentication failed. An HTTP 403 or 429 from the target usually means the exit IP is rate-limited. In that case, rotate the session and retry the request.\n\n## Complete Example: Rotate on Rate Limit\n\nThis script fetches a URL through the proxy. If the target returns 429, it rotates the session once and retries.\n\n```\nimport json\nimport os\nimport time\nimport urllib.request\nfrom urllib.error import HTTPError, URLError\n\nPROXY_URL = (\n    f'http://{os.environ[\"PROXY_USERNAME\"]}:{os.environ[\"PROXY_PASSWORD\"]}'\n    f'@{os.environ[\"PROXY_GATEWAY_HOST\"]}:{os.environ[\"PROXY_HTTP_PORT\"]}'\n)\n\nAPI_KEY = os.environ['ROPROXY_API_KEY']\nSESSION_ID = os.environ['ROPROXY_SESSION_ID']\n\nproxy_handler = urllib.request.ProxyHandler({\n    'http': PROXY_URL,\n    'https': PROXY_URL,\n})\nopener = urllib.request.build_opener(proxy_handler)\n\ndef rotate_session():\n    rotate_url = f'https://backend-api.ro-proxy.com/api/session/{SESSION_ID}/rotate'\n    req = urllib.request.Request(rotate_url, method='POST')\n    req.add_header('Authorization', f'Bearer {API_KEY}')\n    with urllib.request.urlopen(req, timeout=10) as resp:\n        return resp.status\n\ndef fetch_ip():\n    req = urllib.request.Request('https://api.ipify.org?format=json')\n    with opener.open(req, timeout=15) as resp:\n        return json.loads(resp.read().decode())['ip']\n\ndef fetch_with_rotation(url):\n    try:\n        with opener.open(url, timeout=15) as resp:\n            return resp.status, resp.read().decode()\n    except HTTPError as e:\n        if e.code == 429:\n            print('Rate limited. Rotating session...')\n            rotate_session()\n            time.sleep(1)\n            with opener.open(url, timeout=15) as resp:\n                return resp.status, resp.read().decode()\n        raise\n\nif __name__ == '__main__':\n    print('Initial IP:', fetch_ip())\n    status, body = fetch_with_rotation('https://api.ipify.org?format=json')\n    print('Fetch status:', status)\n    print('IP after fetch:', fetch_ip())\n```\n\nThis example uses only the standard library. It verifies the exit IP, rotates on a 429 response, and checks the IP again. In production, add a maximum retry count so a single bad IP does not cause an infinite loop.\n\n## Conclusion\n\nA rotating proxy changes the IP your script presents to the internet, but your client code still controls the request flow, error handling, and retry logic. With `urllib.request`, you get that control without adding dependencies. Configure the proxy handler with your gateway credentials, verify the exit IP with an echo service, and call the rotation endpoint when the target pushes back. Keep credentials in environment variables, handle 407 and 429 explicitly, and retry with a fresh IP when appropriate.\n\nRoProxy gives you sticky, timed, and per-request rotation modes through sessions with individual credentials. Pair those modes with the patterns above, and you can run standard-library Python scripts through a rotating proxy with clear, auditable behavior.\n","https://blog-api.ro-proxy.com/api/blog/posts/using-rotating-proxy-python-urllib/assets",1793605386737]