[{"data":1,"prerenderedAt":20},["ShallowReactive",2],{"blog:post:en:chaos-engineering-proxies-network-fault-injection-ci-cd":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":15,"thumbnail_url":16,"translations":17,"body":18,"asset_base":19},"chaos-engineering-proxies-network-fault-injection-ci-cd","en","Chaos Engineering with Proxies: Injecting Latency, Packet Loss, and Partitions into CI/CD Pipelines","Learn how to use a programmable proxy to simulate real‑world network failures in staging, making your services resilient before they hit production.","2026-10-07",[10,11,12,13,14],"chaos-engineering","proxy","ci-cd","network-fault-injection","resilience",[10,11,12,13,14],"https://blog-api.ro-proxy.com/api/blog/posts/chaos-engineering-proxies-network-fault-injection-ci-cd/thumbnail.svg?lang=en",[5],"## Why simulate network faults with a proxy?\n\nModern distributed systems rarely fail because of code bugs alone. Latency spikes, packet loss, and sudden network partitions are the everyday reality of the internet. Traditional unit tests and even integration tests usually run on a reliable LAN, so they miss these failure modes.\n\nA proxy sits between your test runner and the system under test (SUT). Because it terminates the client connection and opens a new one to the SUT, it can manipulate every byte on the wire: add delay, drop packets, corrupt headers, or even close the connection mid‑stream. Doing this in CI/CD gives you **continuous chaos** — every pull request is exercised against a realistic network.\n\n### Benefits over standalone chaos tools\n\n| Approach | Pros | Cons |\n|---|---|---|\n| **Sidecar chaos daemon (e.g., Chaos Mesh)** | Kubernetes‑native, can affect pods directly | Requires cluster‑level privileges, not portable to local dev |\n| **Application‑level fault injection libraries** | No extra infrastructure | Couples fault logic to code, easy to forget in prod |\n| **Programmable proxy (mitmproxy, Toxiproxy, custom Go/Node proxy)** | Works for any HTTP/HTTPS/SOCKS5 traffic, language‑agnostic, can be dropped into existing test pipelines | Needs a proxy process, but lightweight |\n\nA proxy‑based approach is the sweet spot for teams that already use proxies for scraping, geo‑testing, or TLS inspection — you reuse the same mental model and tooling.\n\n## Choosing a programmable proxy\n\n| Tool | Language | Protocol support | Fault‑injection API |\n|---|---|---|---|\n| **mitmproxy** | Python | HTTP/1.1, HTTP/2, WebSocket, TLS | Inline scripts (`addons`) + `mitmdump --set` |\n| **Toxiproxy** | Go (binary) | TCP (any), HTTP (via upstream) | REST/CLI to create *toxics* (latency, bandwidth, jitter, timeout) |\n| **custom Node.js proxy (http-proxy + net)** | JavaScript | HTTP/HTTPS, WebSocket | Full control in JS, easy to embed in test scripts |\n\nFor a CI/CD pipeline that already runs Docker, **Toxiproxy** is the fastest to spin up: a single container, declarative JSON config, and a tiny client library for Python/Node.\n\n## Spin‑up Toxiproxy in GitHub Actions\n\n```yaml\n# .github/workflows/chaos.yml\nname: Chaos‑Engineering\non: [pull_request]\njobs:\n  chaos-test:\n    runs-on: ubuntu-latest\n    services:\n      toxiproxy:\n        image: ghcr.io/shopify/toxiproxy:2.6\n        ports: [\"8474:8474\", \"8666:8666\"]   # API + example upstream port\n    steps:\n      - uses: actions/checkout@v4\n      - name: Set up Python\n        uses: actions/setup-python@v5\n        with:\n          python-version: '3.11'\n      - name: Install deps\n        run: pip install -r requirements.txt\n      - name: Configure toxic\n        run: |\n          curl -X POST http://localhost:8474/proxies \\\n            -H \"Content-Type: application/json\" \\\n            -d '{\n              \"name\": \"api\",\n              \"listen\": \"0.0.0.0:8666\",\n              \"upstream\": \"api.example.com:443\",\n              \"enabled\": true\n            }'\n          # Add 200 ms latency + 2 % packet loss\n          curl -X POST http://localhost:8474/proxies/api/toxics \\\n            -H \"Content-Type: application/json\" \\\n            -d '{\n              \"name\": \"latency\",\n              \"type\": \"latency\",\n              \"attributes\": {\"latency\": 200, \"jitter\": 50}\n            }'\n          curl -X POST http://localhost:8474/proxies/api/toxics \\\n            -H \"Content-Type: application/json\" \\\n            -d '{\n              \"name\": \"loss\",\n              \"type\": \"limit_data\",\n              \"attributes\": {\"bytes\": 1024}\n            }'\n      - name: Run test suite against proxy\n        env:\n          API_BASE: \"https://localhost:8666\"\n          REQUESTS_CA_BUNDLE: \"${{ runner.temp }}/ca.pem\"   # mitmproxy cert if TLS\n        run: pytest -q tests/chaos/\n```\n\n*Explanation*:\n1. **Service container** runs Toxiproxy.\n2. **Configure step** creates a proxy named `api` that forwards to the real upstream (`api.example.com:443`).\n3. Two **toxics** are added: a latency/jitter toxic and a bandwidth‑limit toxic that effectively drops packets after 1 KB (simulating loss).\n4. Tests point at `https://localhost:8666`; the proxy terminates TLS, so you either trust the Toxiproxy CA or use `REQUESTS_CA_BUNDLE` with a generated cert.\n\n## Writing a Python test that expects faults\n\n```python\n# tests/chaos/test_api_resilience.py\nimport os\nimport time\nimport requests\nimport pytest\n\nAPI_BASE = os.getenv(\"API_BASE\", \"https://localhost:8666\")\n\n\ndef test_get_user_retries_on_transient_failure():\n    \"\"\"Our client should retry idempotent GETs up to 3 times.\"\"\"\n    url = f\"{API_BASE}/v1/users/42\"\n    max_attempts = 3\n    backoff = 0.5\n\n    for attempt in range(1, max_attempts + 1):\n        try:\n            resp = requests.get(url, timeout=2, verify=False)  # verify=False for demo only\n            resp.raise_for_status()\n            data = resp.json()\n            assert data[\"id\"] == 42\n            return  # success\n        except (requests.ConnectTimeout, requests.ReadTimeout, requests.ConnectionError) as exc:\n            if attempt == max_attempts:\n                pytest.fail(f\"All {max_attempts} attempts failed: {exc}\")\n            time.sleep(backoff * attempt)  # exponential backoff\n\n\ndef test_post_order_fails_fast_on_partition():\n    \"\"\"Non‑idempotent POST must not be retried automatically.\"\"\"\n    url = f\"{API_BASE}/v1/orders\"\n    payload = {\"item\": \"widget\", \"qty\": 1}\n    # Toxiproxy can be toggled to \"close\" toxic for a hard partition\n    resp = requests.post(url, json=payload, timeout=2, verify=False)\n    # Expect a 5xx or connection error — the service should surface the fault\n    assert resp.status_code >= 500 or resp.status_code == 0\n```\n\n*Key points*:\n- **Idempotent GET** uses a simple retry loop with exponential backoff.\n- **Non‑idempotent POST** asserts the client *does not* silently retry; the test fails if the client library retries automatically.\n- `verify=False` is only for the demo; in production you would add the Toxiproxy CA to the trust store.\n\n## Extending the fault model\n\n### 1. Simulating a full network partition\n\n```bash\n# Add a \"close\" toxic that aborts the TCP connection after 10 ms\ncurl -X POST http://localhost:8474/proxies/api/toxics \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"name\":\"partition\",\"type\":\"timeout\",\"attributes\":{\"timeout\":10}}'\n```\n\nNow any request that takes longer than 10 ms (which is all of them once latency toxic is present) will be reset, mimicking a split‑brain scenario.\n\n### 2. Corrupting response bodies\n\nToxiproxy doesn’t have a built‑in \"corrupt\" toxic, but you can chain a **mitmproxy** addon after Toxiproxy:\n\n```python\n# addons/corrupt.py\nfrom mitmproxy import http\nimport random\n\nclass CorruptBody:\n    def response(self, flow: http.HTTPFlow) -> None:\n        if random.random() \u003C 0.1:   # 10 % chance\n            flow.response.content = b\"corrupted\"\n            flow.response.headers[\"Content-Length\"] = str(len(flow.response.content))\n\naddons = [CorruptBody()]\n```\n\nRun `mitmdump -s addons/corrupt.py --listen-port 8667 --upstream http://localhost:8666` and point tests at port 8667. This gives you **application‑level corruption** (bad JSON, truncated payloads) on top of transport‑level faults.\n\n### 3. Geo‑specific latency\n\nIf you already use **residential or ISP proxies** for geo‑testing, you can layer Toxiproxy *in front* of those proxies to add region‑specific latency profiles (e.g., 120 ms for EU, 250 ms for APAC). This validates that your client‑side timeout budgets survive real‑world RTT variance.\n\n## Integrating into a broader resilience strategy\n\n| Stage | Tool | What you verify |\n|---|---|---|\n| **Local dev** | `toxiproxy-cli` + `pytest` | Fast feedback, developer can reproduce flaky CI failures locally |\n| **PR gate** | GitHub Actions (as above) | Every change passes a baseline of 200 ms latency + 2 % loss |\n| **Nightly** | Scheduled workflow with higher toxicity (500 ms, 10 % loss, partitions) | Stress‑test circuit breakers, bulkheads, fallback caches |\n| **Production canary** | Sidecar proxy (Envoy + fault filter) on a % of traffic | Real‑user impact measurement before full rollout |\n\n**Circuit‑breaker validation example** (Python `pybreaker`):\n\n```python\nimport pybreaker\nimport requests\n\nbreaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)\n\n@breaker\ndef fetch_user(uid: int):\n    resp = requests.get(f\"{API_BASE}/v1/users/{uid}\", timeout=2, verify=False)\n    resp.raise_for_status()\n    return resp.json()\n\n# In test: hammer the endpoint while latency toxic is active\nfor _ in range(20):\n    try:\n        fetch_user(42)\n    except Exception:\n        pass\nassert breaker.current_state == \"open\"   # breaker should trip\n```\n\n## Clean‑up & security considerations\n\n1. **TLS termination** – Toxiproxy does not terminate TLS by default. For HTTPS upstreams you either:\n   - Run the upstream with a self‑signed cert and trust it in the test runner, **or**\n   - Place a TLS‑terminating proxy (e.g., `mitmproxy` in transparent mode) *in front* of Toxiproxy.\n2. **Secrets** – Never commit real API keys. Use GitHub Actions secrets and inject them as environment variables.\n3. **Resource limits** – Toxiproxy containers are tiny (\u003C10 MB RAM), but a large number of parallel toxics can increase CPU. Set `resources.limits` in the workflow if you run many proxies.\n4. **Determinism** – Use fixed seeds for jitter (`jitter: 0`) when you need reproducible latency numbers; enable jitter only for exploratory runs.\n\n## Checklist before merging\n\n- [ ] All new tests pass under **baseline toxicity** (200 ms + 2 % loss).\n- [ ] Circuit‑breaker, retry, and timeout configs are exercised and logged.\n- [ ] No test disables TLS verification in production code paths.\n- [ ] Documentation updated with the new toxicity profiles and how to run them locally (`make chaos-local`).\n- [ ] Alerting thresholds (e.g., error‑rate > 1 % under toxicity) are reflected in your observability stack.\n\n## TL;DR\n\n1. **Spin up a programmable proxy** (Toxiproxy) as a service container in CI.\n2. **Declare toxics** (latency, bandwidth limit, timeout, close) that map to the failure modes you care about.\n3. **Route your test traffic** through the proxy by setting `API_BASE` or the appropriate proxy env vars.\n4. **Write tests** that assert correct retry, circuit‑breaker, and fallback behavior.\n5. **Graduate** from PR‑gate → nightly → canary → production sidecar for continuous resilience.\n\nBy treating the proxy as a *first‑class chaos injector* you get language‑agnostic, protocol‑agnostic fault injection that lives right next to your existing proxy infrastructure — no extra agents, no kernel modules, just a container and a few API calls. Start small (add 100 ms latency), measure the impact, then expand the toxic suite until your services survive the chaos you *know* they’ll face in the wild.\n","https://blog-api.ro-proxy.com/api/blog/posts/chaos-engineering-proxies-network-fault-injection-ci-cd/assets",1792303106977]