Chaos Engineering with Proxies: Injecting Latency, Packet Loss, and Partitions into CI/CD Pipelines

Why simulate network faults with a proxy?

Modern distributed systems rarely fail because of code bugs alone. Latency spikes, packet loss, and sudden network partitions are the everyday reality of the internet. Traditional unit tests and even integration tests usually run on a reliable LAN, so they miss these failure modes.

A proxy sits between your test runner and the system under test (SUT). Because it terminates the client connection and opens a new one to the SUT, it can manipulate every byte on the wire: add delay, drop packets, corrupt headers, or even close the connection mid‑stream. Doing this in CI/CD gives you continuous chaos — every pull request is exercised against a realistic network.

Benefits over standalone chaos tools

Approach	Pros	Cons
Sidecar chaos daemon (e.g., Chaos Mesh)	Kubernetes‑native, can affect pods directly	Requires cluster‑level privileges, not portable to local dev
Application‑level fault injection libraries	No extra infrastructure	Couples fault logic to code, easy to forget in prod
Programmable proxy (mitmproxy, Toxiproxy, custom Go/Node proxy)	Works for any HTTP/HTTPS/SOCKS5 traffic, language‑agnostic, can be dropped into existing test pipelines	Needs a proxy process, but lightweight

A proxy‑based approach is the sweet spot for teams that already use proxies for scraping, geo‑testing, or TLS inspection — you reuse the same mental model and tooling.

Choosing a programmable proxy

Tool	Language	Protocol support	Fault‑injection API
mitmproxy	Python	HTTP/1.1, HTTP/2, WebSocket, TLS	Inline scripts (addons) + mitmdump --set
Toxiproxy	Go (binary)	TCP (any), HTTP (via upstream)	REST/CLI to create toxics (latency, bandwidth, jitter, timeout)
custom Node.js proxy (http-proxy + net)	JavaScript	HTTP/HTTPS, WebSocket	Full control in JS, easy to embed in test scripts

For a CI/CD pipeline that already runs Docker, Toxiproxy is the fastest to spin up: a single container, declarative JSON config, and a tiny client library for Python/Node.

Spin‑up Toxiproxy in GitHub Actions

# .github/workflows/chaos.yml
name: Chaos‑Engineering
on: [pull_request]
jobs:
  chaos-test:
    runs-on: ubuntu-latest
    services:
      toxiproxy:
        image: ghcr.io/shopify/toxiproxy:2.6
        ports: ["8474:8474", "8666:8666"]   # API + example upstream port
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install deps
        run: pip install -r requirements.txt
      - name: Configure toxic
        run: |
          curl -X POST http://localhost:8474/proxies \
            -H "Content-Type: application/json" \
            -d '{
              "name": "api",
              "listen": "0.0.0.0:8666",
              "upstream": "api.example.com:443",
              "enabled": true
            }'
          # Add 200 ms latency + 2 % packet loss
          curl -X POST http://localhost:8474/proxies/api/toxics \
            -H "Content-Type: application/json" \
            -d '{
              "name": "latency",
              "type": "latency",
              "attributes": {"latency": 200, "jitter": 50}
            }'
          curl -X POST http://localhost:8474/proxies/api/toxics \
            -H "Content-Type: application/json" \
            -d '{
              "name": "loss",
              "type": "limit_data",
              "attributes": {"bytes": 1024}
            }'
      - name: Run test suite against proxy
        env:
          API_BASE: "https://localhost:8666"
          REQUESTS_CA_BUNDLE: "${{ runner.temp }}/ca.pem"   # mitmproxy cert if TLS
        run: pytest -q tests/chaos/

Explanation:

1. Service container runs Toxiproxy.
2. Configure step creates a proxy named api that forwards to the real upstream (api.example.com:443).
3. Two toxics are added: a latency/jitter toxic and a bandwidth‑limit toxic that effectively drops packets after 1 KB (simulating loss).
4. Tests point at https://localhost:8666; the proxy terminates TLS, so you either trust the Toxiproxy CA or use REQUESTS_CA_BUNDLE with a generated cert.

Writing a Python test that expects faults

# tests/chaos/test_api_resilience.py
import os
import time
import requests
import pytest

API_BASE = os.getenv("API_BASE", "https://localhost:8666")


def test_get_user_retries_on_transient_failure():
    """Our client should retry idempotent GETs up to 3 times."""
    url = f"{API_BASE}/v1/users/42"
    max_attempts = 3
    backoff = 0.5

    for attempt in range(1, max_attempts + 1):
        try:
            resp = requests.get(url, timeout=2, verify=False)  # verify=False for demo only
            resp.raise_for_status()
            data = resp.json()
            assert data["id"] == 42
            return  # success
        except (requests.ConnectTimeout, requests.ReadTimeout, requests.ConnectionError) as exc:
            if attempt == max_attempts:
                pytest.fail(f"All {max_attempts} attempts failed: {exc}")
            time.sleep(backoff * attempt)  # exponential backoff


def test_post_order_fails_fast_on_partition():
    """Non‑idempotent POST must not be retried automatically."""
    url = f"{API_BASE}/v1/orders"
    payload = {"item": "widget", "qty": 1}
    # Toxiproxy can be toggled to "close" toxic for a hard partition
    resp = requests.post(url, json=payload, timeout=2, verify=False)
    # Expect a 5xx or connection error — the service should surface the fault
    assert resp.status_code >= 500 or resp.status_code == 0

Key points:

• Idempotent GET uses a simple retry loop with exponential backoff.
• Non‑idempotent POST asserts the client does not silently retry; the test fails if the client library retries automatically.
• verify=False is only for the demo; in production you would add the Toxiproxy CA to the trust store.

Extending the fault model

1. Simulating a full network partition

# Add a "close" toxic that aborts the TCP connection after 10 ms
curl -X POST http://localhost:8474/proxies/api/toxics \
  -H "Content-Type: application/json" \
  -d '{"name":"partition","type":"timeout","attributes":{"timeout":10}}'

Now any request that takes longer than 10 ms (which is all of them once latency toxic is present) will be reset, mimicking a split‑brain scenario.

2. Corrupting response bodies

Toxiproxy doesn’t have a built‑in "corrupt" toxic, but you can chain a mitmproxy addon after Toxiproxy:

# addons/corrupt.py
from mitmproxy import http
import random

class CorruptBody:
    def response(self, flow: http.HTTPFlow) -> None:
        if random.random() < 0.1:   # 10 % chance
            flow.response.content = b"corrupted"
            flow.response.headers["Content-Length"] = str(len(flow.response.content))

addons = [CorruptBody()]

Run mitmdump -s addons/corrupt.py --listen-port 8667 --upstream http://localhost:8666 and point tests at port 8667. This gives you application‑level corruption (bad JSON, truncated payloads) on top of transport‑level faults.

3. Geo‑specific latency

If you already use residential or ISP proxies for geo‑testing, you can layer Toxiproxy in front of those proxies to add region‑specific latency profiles (e.g., 120 ms for EU, 250 ms for APAC). This validates that your client‑side timeout budgets survive real‑world RTT variance.

Integrating into a broader resilience strategy

Stage	Tool	What you verify
Local dev	toxiproxy-cli + pytest	Fast feedback, developer can reproduce flaky CI failures locally
PR gate	GitHub Actions (as above)	Every change passes a baseline of 200 ms latency + 2 % loss
Nightly	Scheduled workflow with higher toxicity (500 ms, 10 % loss, partitions)	Stress‑test circuit breakers, bulkheads, fallback caches
Production canary	Sidecar proxy (Envoy + fault filter) on a % of traffic	Real‑user impact measurement before full rollout

Circuit‑breaker validation example (Python pybreaker):

import pybreaker
import requests

breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)

@breaker
def fetch_user(uid: int):
    resp = requests.get(f"{API_BASE}/v1/users/{uid}", timeout=2, verify=False)
    resp.raise_for_status()
    return resp.json()

# In test: hammer the endpoint while latency toxic is active
for _ in range(20):
    try:
        fetch_user(42)
    except Exception:
        pass
assert breaker.current_state == "open"   # breaker should trip

Clean‑up & security considerations

1. TLS termination – Toxiproxy does not terminate TLS by default. For HTTPS upstreams you either:
   • Run the upstream with a self‑signed cert and trust it in the test runner, or
   • Place a TLS‑terminating proxy (e.g., mitmproxy in transparent mode) in front of Toxiproxy.
2. Secrets – Never commit real API keys. Use GitHub Actions secrets and inject them as environment variables.
3. Resource limits – Toxiproxy containers are tiny (<10 MB RAM), but a large number of parallel toxics can increase CPU. Set resources.limits in the workflow if you run many proxies.
4. Determinism – Use fixed seeds for jitter (jitter: 0) when you need reproducible latency numbers; enable jitter only for exploratory runs.

Checklist before merging

• All new tests pass under baseline toxicity (200 ms + 2 % loss).
• Circuit‑breaker, retry, and timeout configs are exercised and logged.
• No test disables TLS verification in production code paths.
• Documentation updated with the new toxicity profiles and how to run them locally (make chaos-local).
• Alerting thresholds (e.g., error‑rate > 1 % under toxicity) are reflected in your observability stack.

TL;DR

1. Spin up a programmable proxy (Toxiproxy) as a service container in CI.
2. Declare toxics (latency, bandwidth limit, timeout, close) that map to the failure modes you care about.
3. Route your test traffic through the proxy by setting API_BASE or the appropriate proxy env vars.
4. Write tests that assert correct retry, circuit‑breaker, and fallback behavior.
5. Graduate from PR‑gate → nightly → canary → production sidecar for continuous resilience.

By treating the proxy as a first‑class chaos injector you get language‑agnostic, protocol‑agnostic fault injection that lives right next to your existing proxy infrastructure — no extra agents, no kernel modules, just a container and a few API calls. Start small (add 100 ms latency), measure the impact, then expand the toxic suite until your services survive the chaos you know they’ll face in the wild.