Using Proxies to Validate Zero‑Trust Network Policies in Kubernetes

Introduction

Zero‑trust security assumes that no network segment is inherently trusted. In Kubernetes this often translates to policies in Envoy or Istio that require JWT tokens, source IP ranges, or mutual TLS before allowing a request to reach a service. Testing these policies reliably means you need to generate traffic that appears to come from outside the cluster, from various IP addresses, and sometimes with specific headers. Rotating proxies give you a convenient way to emulate such external clients while keeping your test suite deterministic and repeatable.

Why Use Proxies for Zero‑Trust Validation

• IP diversity – A proxy pool can supply addresses from many countries, letting you verify geo‑based restrictions.
• Rotation on demand – You can change the source IP for each request to simulate brute‑force or credential‑stuffing attempts.
• Isolation – The application code stays unchanged; all traffic redirection happens at the sidecar level.
• Cost‑effective – Compared with spinning up real VMs in multiple clouds, a proxy service is cheaper and easier to manage.

Architecture Overview

Your workload runs in a pod with an Envoy sidecar. The sidecar is configured to forward outbound traffic through an HTTP proxy before reaching the target service. The proxy rotates its outgoing IP address according to the credentials you provide. The flow looks like:

Pod (app) → Envoy sidecar (listener) → HTTP proxy → Target service (e.g., httpbin.org/ip)

Because the proxy sits between Envoy and the destination, Envoy sees the proxy as the immediate peer, but the final destination sees the proxy’s IP. This lets you test policies that depend on the client IP observed by the service.

Prerequisites

• A Kubernetes cluster (v1.22+)
• kubectl configured to access the cluster
• Istio installed or the ability to inject a raw Envoy sidecar via a pod annotation
• Access to a rotating residential or datacenter proxy provider that offers HTTP/HTTPS endpoints with username/password authentication
• curl and a simple test service (we will use the public httpbin.org/ip endpoint)

Step‑by‑Step Guide

1. Deploy a Test Service

We will deploy a tiny echo service that returns the request headers and source IP as seen by the service. Save this as test-service.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - name: echo
        image: hashicorp/http-echo
        args:
        - "-text=hello from echo"
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  selector:
    app: echo
  ports:
  - port: 80
    targetPort: 5678
    protocol: TCP

Apply it:

kubectl apply -f test-service.yaml

Wait for the pod to be running.

2. Inject Envoy Sidecar

If you already have Istio with automatic sidecar injection, label the namespace:

kubectl label namespace default istio-injection=enabled

Otherwise you can manually add a sidecar container. Below is a minimal Envoy configuration that forwards all outbound traffic to an HTTP proxy. Save it as envoy-sidecar.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: echo-with-proxy
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  containers:
  - name: echo
    image: hashicorp/http-echo
    args:
    - "-text=hello from echo"
    ports:
    - containerPort: 5678
  - name: envoy
    image: envoyproxy/envoy:v1.25-latest
    command: ["/usr/local/bin/envoy"]
    args:
    - "-c"
    - "/etc/envoy/envoy.yaml"
    - "--service-cluster"
    - "echo-svc"
    - "--service-node"
    - "echo-node"
    volumeMounts:
    - name: envoy-config
      mountPath: /etc/envoy
      readOnly: true
    - name: proxy-auth
      mountPath: /etc/proxy
      readOnly: true
  volumes:
  - name: envoy-config
    configMap:
      name: envoy-config
  - name: proxy-auth
    secret:
      secretName: proxy-cred

We will create the ConfigMap and Secret in the next steps.

3. Configure Envoy to Forward Traffic via HTTP Proxy

Create a ConfigMap named envoy-config with the following Envoy YAML (note the use of single quotes for strings inside the JSON‑like structure – Envoy accepts YAML, so we keep double quotes only where required by the YAML syntax; we avoid extra double quotes in explanatory text):

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 15001
    filter_chains:
    - filters:
    - name: envoy.filters.network.http_connection_manager
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        stat_prefix: ingress_http
        route_config:
          name: local_route
          virtual_hosts:
          - name: backend
            domains: ["*"]
            routes:
            - match:
                prefix: "/"
              route:
                cluster: proxy_cluster
        http_filters:
    - name: envoy.filters.router
  clusters:
  - name: proxy_cluster
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: proxy_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: proxy.example.com  # replace with your proxy host
                port_value: 8000
    # Optional TLS upstream if your proxy requires it
    # transport_socket:
    #   name: envoy.transport_sockets.tls
    #   typed_config:
    #     "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext

Replace proxy.example.com and port with your provider’s endpoint.

Create the ConfigMap:

kubectl create configmap envoy-config --from-file=envoy.yaml=<path-to-above-yaml>

4. Set Up Rotating Proxy Credentials

Most providers give you a username and password (or token) that you must include in the Proxy‑Authorization header. Create a Kubernetes Secret named proxy-cred:

kubectl create secret generic proxy-cred \
  --from-literal=username=YOUR_USERNAME \
  --from-literal=password=YOUR_PASSWORD

Now we need Envoy to add this header to every outbound request. Extend the listener configuration with a request_headers_to_add entry. Add the following under http_filters in the ConfigMap (still YAML, no extra double quotes in comments):

        http_filters:
        - name: envoy.filters.router
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.router.v3.Router
        - name: envoy.filters.headers
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.headers.v3.HeaderModifier
            request_headers_to_add:
            - header:
                key: "Proxy-Authorization"
                value: "Basic "
                append: false

The actual Base64‑encoded username:password must be computed and placed in the value. You can do this once and store the result in the secret, then have Envoy read it via a header value plugin, but for simplicity we pre‑compute:

printf "YOUR_USERNAME:YOUR_PASSWORD" | base64

Suppose the output is eW91cl91c2VybmFtZTpZb3VyX3Bhc3N3b3Jk. Update the ConfigMap so the header value is Basic eW91cl91c2VybmFtZTpZb3VyX3Bhc3N3b3Jk.

5. Apply Configuration and Test

Deploy the pod with the sidecar:

kubectl apply -f envoy-sidecar.yaml

Wait for both containers to be Ready. Then open a shell in the echo container:

kubectl exec -it echo-with-proxy -c echo -- sh

From inside the container, call the echo service via localhost:15001 (Envoy listener) which will forward through the proxy to httpbin.org/ip:

curl -s http://localhost:15001/https://httpbin.org/ip

You should see a JSON response like:

{
  "origin": "203.0.113.45"
}

Run the command a few times; the IP address should change if your proxy pool rotates per request or per connection.

6. Validate Zero‑Trust Policy

Now suppose you have an Envoy RBAC policy that only allows inbound traffic to the echo service from the IP range 10.0.0.0/8 (simulating an internal trusted network). The policy might look like this in an Istio AuthorizationPolicy:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: echo-internal-only
spec:
  selector:
    matchLabels:
      app: echo
  action: DENY
  rules:
  - when:
    - key: source.ip
      notInRanges: ["10.0.0.0/8"]

Apply this policy. Then, from within the pod, try to reach the echo service via the proxy (which will appear as an external IP). The request should be denied with a 403 response.

curl -s -o /dev/null -w "%{http_code}" http://localhost:15001/http://echo:80/hello

You should see 403. Now temporarily add a proxy IP that falls inside 10.0.0.0/8 (some providers let you select datacenter IPs in specific ranges) or configure the proxy to use a static IP for a test. If the policy is working, the request will succeed (200). This demonstrates that your zero‑trust rule is enforced correctly.

7. Automate Rotation Detection

To continuously verify that the proxy is rotating, you can run a small Python script inside the pod:

import time
import requests

PROXY = "http://proxy.example.com:8000"
AUTH = "Basic eW91cl91c2VybmFtZTpZb3VyX3Bhc3N3b3Jk"

for i in range(10):
    proxies = {"http": PROXY, "https": PROXY}
    headers = {"Proxy-Authorization": AUTH}
    try:
        r = requests.get("http://httpbin.org/ip", proxies=proxies, headers=headers, timeout=5)
        print(f"Iter {i}: {r.json()}")
    except Exception as e:
        print(f"Error: {e}")
    time.sleep(2)

Run it with python3 script.py. You should see different origin values appearing over time.

Real‑World Example: Financial‑Services API

A fintech company needed to verify that their transaction API blocked requests originating from known malicious IP ranges while allowing legitimate partners. They deployed the API behind an Envoy sidecar in a test cluster, configured the sidecar to use a rotating residential proxy pool, and ran a CI job that:

1. Cycled through a list of known bad IPs (provided by the proxy’s geo‑targeting feature).
2. Asserted each request returned HTTP 403.
3. Switched to a set of whitelisted partner IPs and asserted HTTP 200.

Because the proxy rotation was automated, the test suite could be run nightly against fresh IP addresses, giving confidence that the zero‑trust policy stayed up‑to‑date as the threat landscape changed.

Troubleshooting

• Proxy authentication failures – Ensure the Base64 string in the Proxy-Authorization header is correct. A common mistake is forgetting the Basic prefix or adding extra whitespace.
• TLS handshake errors – If your proxy requires HTTPS, add a transport_socket of type tls to the proxy_cluster and provide the necessary root certificates via a secret mounted into Envoy.
• DNS leaks – Envoy will resolve the proxy host name inside the pod. Make sure the pod’s /etc/resolv.conf points to a DNS server you trust; otherwise the proxy hostname could be resolved locally, exposing your intent.
• Empty responses – Verify that the listener port (15001 in the example) matches the container port you are curling. Use kubectl logs <pod> -c envoy to see Envoy’s error logs.

Best Practices

• Keep the proxy pool size adequate for your test concurrency; otherwise you may see HTTP 429 (Too Many Requests) from the proxy provider.
• Monitor proxy health via the provider’s API or by periodically checking the returned IP; replace unhealthy nodes automatically.
• If you need sticky sessions (e.g., to maintain login state), configure the proxy to retain the same IP for a configurable duration and set Envoy’s max_requests_per_connection accordingly.
• Use a sidecar metric export (Envoy stats) to capture request counts, response codes, and latency; plug these into Prometheus/Grafana for observability.
• Rotate credentials regularly and store them as Kubernetes Secrets, never hard‑code them in manifests.

Conclusion

By placing a rotating HTTP proxy between Envoy sidecars and your services, you gain a flexible, low‑cost way to emulate external traffic sources for zero‑trust policy validation. The approach requires no changes to your application code, works with any language or framework, and integrates naturally into CI/CD pipelines. With the steps outlined above you can confidently test IP‑based allow/deny lists, geo‑restrictions, and other network‑level controls before they reach production.