RoProxy
Back to all posts
Using a Rotating Proxy with Python's urllib.request
python proxy urllib rotation

Using a Rotating Proxy with Python's urllib.request

November 2, 2026

Many scraping and automation scripts start with the requests library, but Python's standard library includes urllib.request, which works without installing anything. That matters in locked-down build images, AWS Lambda layers, or internal tools where adding dependencies is slow or blocked. A rotating proxy sits between your script and the target, swapping the exit IP on a schedule or per connection. This guide shows how to route urllib.request through a rotating proxy, add authentication, verify the exit IP, and rotate the IP between requests using only built-in modules.

Why Use urllib.request with a Rotating Proxy

urllib.request is part of Python's standard library. It supports HTTP and HTTPS, proxy handlers, and basic authentication out of the box. When you pair it with a rotating proxy, each request can appear to come from a different IP address. This helps avoid rate limits, access geo-restricted content, and distribute load across multiple exit points.

The main trade-off is convenience. urllib.request is more verbose than requests or httpx. It does not automatically decode JSON, manage sessions, or retry on connection errors. You write that logic yourself. For a focused integration with a rotating proxy, that verbosity is actually useful: it makes exactly when and how the proxy is used explicit in your code.

How the Proxy Connection Works

A forward proxy accepts a request from your client, connects to the destination on your behalf, and returns the response. For HTTPS, your client sends a CONNECT request through the proxy to establish a tunnel, then performs TLS directly with the target. The proxy sees the destination host but not the encrypted payload.

Your proxy session provides a gateway host and port. In code, the proxy URL looks like this:

http://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT

Use the same URL for both http and https keys. The scheme inside the proxy URL remains http even when the final destination is HTTPS, because you are talking to the proxy over plain HTTP and asking it to open a tunnel.

If your plan includes a SOCKS5 port, urllib.request does not support SOCKS5 natively. You would need an external wrapper or a different library. This guide uses the HTTP port, which works for both HTTP and HTTPS targets.

Do not mix proxied and direct requests through the same opener unless you intend to. If part of your script should call internal APIs directly, create a second opener without the ProxyHandler and use it for those calls.

Configure the Proxy Handler

Create a ProxyHandler and pass it to build_opener. The opener replaces the default URL opener so every request you make through it uses the proxy.

import urllib.request

proxy_url = 'http://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT'

proxy_handler = urllib.request.ProxyHandler({
    'http': proxy_url,
    'https': proxy_url,
})

opener = urllib.request.build_opener(proxy_handler)

If you use RoProxy, each session has its own username and password. Insert those credentials into USERNAME and PASSWORD. The gateway host and HTTP port come from your session details. Sessions on Pro and Business plans also expose a SOCKS5 port, but this example uses the HTTP port.

Install the opener globally if you want every urllib.request.urlopen call in the process to use the proxy:

urllib.request.install_opener(opener)

Otherwise, call opener.open(request) directly for scoped proxy use.

Add Basic Authentication

The username and password inside the proxy URL handle proxy authentication. If the proxy rejects them, you receive an HTTP 407 error. Do not hardcode credentials in version control. Read them from environment variables:

import os

username = os.environ['PROXY_USERNAME']
password = os.environ['PROXY_PASSWORD']
gateway_host = os.environ['PROXY_GATEWAY_HOST']
http_port = os.environ['PROXY_HTTP_PORT']

proxy_url = f'http://{username}:{password}@{gateway_host}:{http_port}'

If the password contains special characters, percent-encode it with urllib.parse.quote before inserting it into the URL.

Test the Exit IP

Before rotating anything, confirm your traffic actually leaves through the proxy. Request an IP echo service and compare the result to your local address.

import json

request = urllib.request.Request('https://api.ipify.org?format=json')

with opener.open(request, timeout=10) as response:
    data = json.loads(response.read().decode())
    print('Exit IP:', data['ip'])

Run this twice with different proxy sessions. If the IP changes, the proxy is working. If it matches your local IP, the proxy settings were ignored or bypassed.

Rotate the IP Between Requests

Rotation behavior depends on the session mode:

  • Sticky: Keeps the same IP until it rotates, fails, or you trigger a change. Use this for multi-page flows where the IP must stay constant.
  • Timed: Changes the IP after an interval you set. Use this for long-running jobs that need a fresh address periodically.
  • Per request: A new IP for every new connection. Available on the Business plan only. Use this for stateless checks where each request should appear independent.

urllib.request does not rotate IPs itself. Rotation happens at the proxy layer. Your code controls when to ask for a new IP by calling the rotation endpoint.

import urllib.request

rotate_url = 'https://backend-api.ro-proxy.com/api/session/SESSION_ID/rotate'

request = urllib.request.Request(rotate_url, method='POST')
request.add_header('Authorization', 'Bearer API_KEY')

with urllib.request.urlopen(request, timeout=10) as response:
    print('Rotation status:', response.status)

Replace SESSION_ID with the identifier for the session you want to rotate. A successful response means the session now uses a different IP for new connections.

Handle Errors and Retries

Proxy connections fail for reasons outside your control: the target blocks the exit IP, the proxy gateway is unreachable, or authentication expires. Wrap proxy calls in try/except blocks and distinguish between proxy errors and target errors.

from urllib.error import URLError, HTTPError

try:
    with opener.open(request, timeout=15) as response:
        body = response.read().decode()
except HTTPError as e:
    print(f'Target returned {e.code}: {e.reason}')
except URLError as e:
    print(f'Proxy or network failure: {e.reason}')

An HTTP 407 means proxy authentication failed. An HTTP 403 or 429 from the target usually means the exit IP is rate-limited. In that case, rotate the session and retry the request.

Complete Example: Rotate on Rate Limit

This script fetches a URL through the proxy. If the target returns 429, it rotates the session once and retries.

import json
import os
import time
import urllib.request
from urllib.error import HTTPError, URLError

PROXY_URL = (
    f'http://{os.environ["PROXY_USERNAME"]}:{os.environ["PROXY_PASSWORD"]}'
    f'@{os.environ["PROXY_GATEWAY_HOST"]}:{os.environ["PROXY_HTTP_PORT"]}'
)

API_KEY = os.environ['ROPROXY_API_KEY']
SESSION_ID = os.environ['ROPROXY_SESSION_ID']

proxy_handler = urllib.request.ProxyHandler({
    'http': PROXY_URL,
    'https': PROXY_URL,
})
opener = urllib.request.build_opener(proxy_handler)

def rotate_session():
    rotate_url = f'https://backend-api.ro-proxy.com/api/session/{SESSION_ID}/rotate'
    req = urllib.request.Request(rotate_url, method='POST')
    req.add_header('Authorization', f'Bearer {API_KEY}')
    with urllib.request.urlopen(req, timeout=10) as resp:
        return resp.status

def fetch_ip():
    req = urllib.request.Request('https://api.ipify.org?format=json')
    with opener.open(req, timeout=15) as resp:
        return json.loads(resp.read().decode())['ip']

def fetch_with_rotation(url):
    try:
        with opener.open(url, timeout=15) as resp:
            return resp.status, resp.read().decode()
    except HTTPError as e:
        if e.code == 429:
            print('Rate limited. Rotating session...')
            rotate_session()
            time.sleep(1)
            with opener.open(url, timeout=15) as resp:
                return resp.status, resp.read().decode()
        raise

if __name__ == '__main__':
    print('Initial IP:', fetch_ip())
    status, body = fetch_with_rotation('https://api.ipify.org?format=json')
    print('Fetch status:', status)
    print('IP after fetch:', fetch_ip())

This example uses only the standard library. It verifies the exit IP, rotates on a 429 response, and checks the IP again. In production, add a maximum retry count so a single bad IP does not cause an infinite loop.

Conclusion

A rotating proxy changes the IP your script presents to the internet, but your client code still controls the request flow, error handling, and retry logic. With urllib.request, you get that control without adding dependencies. Configure the proxy handler with your gateway credentials, verify the exit IP with an echo service, and call the rotation endpoint when the target pushes back. Keep credentials in environment variables, handle 407 and 429 explicitly, and retry with a fresh IP when appropriate.

RoProxy gives you sticky, timed, and per-request rotation modes through sessions with individual credentials. Pair those modes with the patterns above, and you can run standard-library Python scripts through a rotating proxy with clear, auditable behavior.