[{"data":1,"prerenderedAt":22},["ShallowReactive",2],{"blog:post:en:automated-retrieval-of-web-archives-with-rotating-proxies":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":15,"thumbnail_url":18,"translations":19,"body":20,"asset_base":21},"automated-retrieval-of-web-archives-with-rotating-proxies","en","Automated Retrieval of Web Archives with Rotating Proxies","Systematically fetch archived snapshots using rotating proxies, handle rate limits, and validate content with Python code.","2026-11-04",[10,11,12,13,14],"rotating proxies","web archives","python","scraping","automation",[16,17,12,13,14],"rotating-proxies","web-archives","https://blog-api.ro-proxy.com/api/blog/posts/automated-retrieval-of-web-archives-with-rotating-proxies/thumbnail.svg?lang=en",[5],"## Overview\n\nArchiving platforms such as the Wayback Machine preserve historical versions of web pages, but direct access can be throttled or blocked by target sites. By combining rotating proxy sessions with a disciplined fetching strategy, you can retrieve large batches of archived snapshots reliably, respect rate limits, and verify that each snapshot is still accessible.\n\n## Why Archive Scraping Matters\n\n- **Historical research:** Scholars need consistent captures of pages that have changed or disappeared.\n- **Content audits:** Marketing teams compare past and present messaging across domains.\n- **Compliance checks:** Organizations verify that public disclosures match archived records.\n- **Data enrichment:** AI training sets benefit from a temporal dimension.\n\nA robust pipeline must rotate IPs to avoid detection, throttle requests to stay within polite limits, and confirm that each snapshot is reachable from a variety of geographic locations.\n\n## Setting Up a Rotating Proxy Session with RoProxy\n\nRoProxy provides rotating sessions that can be controlled via a simple REST API. The workflow below assumes you already have a Pro or Business plan session with a known **SESSION_ID** and an **API_KEY**.\n\n### Step 1 – Obtain a Fresh IP\n\n```python\nimport os\nimport requests\nimport time\n\nAPI_KEY = os.getenv(\"API_KEY\", \"API_KEY\")\nSESSION_ID = os.getenv(\"SESSION_ID\", \"SESSION_ID\")\n\nbase = \"https://backend-api.ro-proxy.com\"\nheaders = {\"Authorization\": f\"Bearer {API_KEY}\"}\n\n# List existing sessions (optional)\nsessions_resp = requests.get(f\"{base}/api/session\", headers=headers)\nsessions = sessions_resp.json()  # expects a list of session objects\n\n# Rotate the chosen session\nrotate_url = f\"{base}/api/session/{SESSION_ID}/rotate\"\nrotate_resp = requests.post(rotate_url, headers=headers)\nprint(f\"Rotation response: {rotate_resp.status_code}\")\n```\n\nThe call to `/api/session/SESSION_ID/rotate` forces an immediate IP change, which is useful before a high‑volume crawl.\n\n### Step 2 – Configure Proxy Environment Variables\n\nReplace the placeholders with your actual gateway details. RoProxy sessions expose both an HTTP port and, on Pro/Business plans, a SOCKS5 port.\n\n```bash\n# HTTP proxy (used for both http and https requests)\nexport HTTP_PROXY=\"http://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT\"\nexport HTTPS_PROXY=\"http://USERNAME:PASSWORD@GATEWAY_HOST:HTTP_PORT\"\n\n# SOCKS5 proxy (optional, Pro/Business)\nexport SOCKS5_PROXY=\"socks5://USERNAME:PASSWORD@GATEWAY_HOST:SOCKS5_PORT\"\n```\n\nThese variables are read automatically by libraries such as `requests` when a proxy dict is supplied.\n\n## Building the Archive Fetcher in Python\n\nBelow is a self‑contained script that demonstrates the full flow: session rotation, proxy configuration, batch retrieval of archived URLs, rate limiting, and integrity verification.\n\n### Step 3 – Core Archive Retrieval Function\n\n```python\nimport os\nimport requests\nimport hashlib\nimport json\nimport time\n\n# Load proxy settings from environment variables\nhttp_proxy = os.getenv(\"HTTP_PROXY\")\nsocks5_proxy = os.getenv(\"SOCKS5_PROXY\")\n\nproxies = {\n    \"http\": http_proxy,\n    \"https\": http_proxy,\n}\nif socks5_proxy:\n    # Some libraries support \"socks5\" scheme; requests does not natively support it.\n    # For demonstration we keep the HTTP proxy; switch to urllib or aiohttp if needed.\n    pass\n\ndef fetch_archive_snapshot(original_url: str, timestamp: str) -> bytes | None:\n    \"\"\"Return raw HTML of an archived snapshot or None on failure.\"\"\"\n    archive_url = f\"https://web.archive.org/web/{timestamp}/{original_url}\"\n    try:\n        resp = requests.get(archive_url, proxies=proxies, timeout=15)\n        if resp.status_code == 200:\n            return resp.content\n        else:\n            print(f\"  ↳ {archive_url} → status {resp.status_code}\")\n            return None\n    except Exception as e:\n        print(f\"  ↳ {archive_url} → error {e}\")\n        return None\n```\n\n### Step 4 – Rate Limiting and Batch Processing\n\n```python\ndef crawl_archives(urls, timestamps, out_path=\"archives.json\"):\n    results = []\n    for idx, (url, ts) in enumerate(zip(urls, timestamps), start=1):\n        print(f\"Processing {idx}/{len(urls)}: {url} @ {ts}\")\n        content = fetch_archive_snapshot(url, ts)\n        entry = {\n            \"original_url\": url,\n            \"timestamp\": ts,\n            \"archive_url\": f\"https://web.archive.org/web/{ts}/{url}\",\n            \"content_hash\": hashlib.sha256(content).hexdigest() if content else None,\n            \"size_bytes\": len(content) if content else 0,\n            \"retrieved_at\": time.isoformat(time.localtime()),\n        }\n        results.append(entry)\n        # Respect a conservative throttle – 1 request per second per IP rotation\n        time.sleep(1)\n\n    with open(out_path, \"w\", encoding=\"utf-8\") as f:\n        json.dump(results, f, indent=2)\n    print(f\"Saved {len(results)} entries to {out_path}\")\n```\n\nThe script rotates the RoProxy session before the crawl (Step 1) and then respects a one‑request‑per‑second ceiling to stay under typical archival limits.\n\n### Step 5 – Verifying Exit IPs and Geographic Diversity\n\nTo ensure traffic originates from distinct IPs, you can query a public service and compare results across runs.\n\n```python\ndef check_exit_ip() -> str:\n    \"\"\"Return the public IP seen by the proxy.\"\"\"\n    resp = requests.get(\"https://api.ipify.org?format=json\", proxies=proxies, timeout=10)\n    data = resp.json()\n    return data[\"ip\"]\n\n# Example usage after a rotation\nprint(f\"Current exit IP: {check_exit_ip()}\")\n```\n\nIf you need to confirm geographic variety, combine this check with a GeoIP lookup (e.g., MaxMind DB) or simply rotate sessions for each country you target on a Pro/Business plan.\n\n## Scheduling and Scaling the Job\n\nA typical archive crawl can involve thousands of URLs. Use a scheduler like `cron` or `systemd timers` to invoke the Python script periodically. For higher throughput, consider:\n\n- **Concurrency:** Replace `time.sleep` with an asynchronous framework (e.g., `asyncio` + `aiohttp`) and limit concurrent tasks to avoid overwhelming the proxy.\n- **Session pooling:** Reuse the same session ID across multiple runs and only rotate when a failure threshold is reached.\n- **Error back‑off:** Implement exponential back‑off for HTTP 429 or 503 responses.\n\nExample snippet for a simple `cron` job (run daily at 02:00):\n\n```bash\n0 2 * * * /usr/bin/python3 /path/to/archive_crawler.py >> /var/log/archive_crawl.log 2>&1\n```\n\n## Best Practices and Troubleshooting\n\n- **Proxy health:** Periodically call the RoProxy session list endpoint to confirm the session is still active. A 401 or 404 indicates the credentials need refresh.\n- **CAPTCHA handling:** If the target site presents a CAPTCHA, rotate the IP immediately using the same `/rotate` endpoint and retry the request.\n- **Logging:** Store rotation timestamps and exit IPs in a log file; this helps correlate blocks with specific IP changes.\n- **Fallback proxies:** Keep a secondary proxy configuration (e.g., a backup HTTP endpoint) ready for when the primary gateway becomes unreachable.\n- **SSL verification:** For HTTPS archives, disable certificate verification only in controlled environments; otherwise, keep it enabled to avoid man‑in‑the‑middle risks.\n\nWhen debugging connection timeouts, use tools like `tcpdump` or Wireshark to inspect the TCP handshake. A missing TCP SYN‑ACK often points to a blocked or overloaded gateway.\n\n## Conclusion\n\nAutomated archive retrieval becomes dependable when you combine disciplined IP rotation with careful rate limiting and thorough verification. By leveraging RoProxy’s session API and configuring its gateways through standard proxy environment variables, you obtain a clean, reproducible pipeline that can scale from a handful of historic pages to millions of snapshots. Remember to rotate sessions before heavy bursts, monitor exit IPs, and keep fallback mechanisms in place to ensure your archival data stays current and complete.\n\nWith RoProxy you can focus on the content you need rather than the mechanics of staying under anti‑scraping defenses.\n","https://blog-api.ro-proxy.com/api/blog/posts/automated-retrieval-of-web-archives-with-rotating-proxies/assets",1793951898769]