The first time a service stopped resolving in one of my clusters, I spent an evening reading iptables chains. Hundreds of rules, generated by kube-proxy, evaluated top to bottom. I never found the actual problem. I restarted a node and it went away. That bothered me more than the outage did. I was running something I couldn’t read.

That feeling is why I moved to Cilium. It uses eBPF to push networking logic down into the Linux kernel and skips iptables entirely. You get better performance, you can actually see what your traffic is doing, and network policies stop being a guessing game.

This is what I run in my clusters now. I’ll build it up from the simplest piece, so you can stop reading at whatever depth you need and still walk away with something working.

What is eBPF?

Start here, because the rest only makes sense once this clicks. eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel without patching kernel source or loading a kernel module.

flowchart TD
    subgraph userspace["User Space"]
        APP["Application"]
        CILIUM["Cilium Agent"]
    end

    subgraph kernel["Kernel Space"]
        EBPF["eBPF Programs"]
        NET["Network Stack"]
        HOOKS["Kernel Hooks<br/>(XDP, TC, Socket)"]
    end

    APP --> NET
    CILIUM -->|"loads"| EBPF
    EBPF --> HOOKS
    HOOKS --> NET

Traditional networking leans on iptables, a chain of rules walked one by one until something matches. eBPF programs get compiled and run directly in the kernel, and they look things up in maps instead of scanning a list. That sounds like a small distinction until you scale it.

Here’s where it bites. Iptables with 10,000 services means a linear scan that gets slower with every service you add. Cilium with 10,000 services does a constant-time map lookup. Same workload, very different behaviour under load. And when something is off, I can inspect the maps directly instead of guessing which of a few hundred rules misfired.

Installing Cilium

The minimum viable version: swap your existing CNI for Cilium and check that it came up.

# Install Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz

# Install Cilium in cluster
cilium install --version 1.15.0

# Verify installation
cilium status --wait

The CLI is fine for a first look, but I don’t run anything by hand twice. Here’s the same install as an ArgoCD Application, so the cluster’s network layer lives in Git like everything else:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cilium
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://helm.cilium.io/
    chart: cilium
    targetRevision: 1.15.0
    helm:
      values: |
        kubeProxyReplacement: true
        k8sServiceHost: "10.0.0.1"
        k8sServicePort: "6443"

        hubble:
          enabled: true
          relay:
            enabled: true
          ui:
            enabled: true

        operator:
          replicas: 2

        ipam:
          mode: kubernetes

  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system

kube-proxy Replacement

This is the first layer worth turning on. Cilium can replace kube-proxy entirely and implement Kubernetes Services in eBPF, which is how I got rid of those iptables chains that started this whole thing:

# Cilium values
kubeProxyReplacement: true
k8sServiceHost: "10.0.0.1"  # API server IP
k8sServicePort: "6443"

What you get:

  • No iptables rules for services
  • Direct server return for better performance
  • Socket-level load balancing, where the decision happens at connect() instead of on every packet
  • Maglev hashing for consistent backend selection

Verify it’s working:

kubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxyReplacement
# KubeProxyReplacement: True [eth0 10.0.0.10 (Direct Routing)]

Network Policies

Now the part that actually changes how you think about security. Cilium speaks standard Kubernetes NetworkPolicy, and it adds its own CiliumNetworkPolicy on top with a lot more reach. Start with the standard one if you’ve never written a policy before.

Kubernetes NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

CiliumNetworkPolicy (Extended)

Once the basics make sense, the native policy is where it gets interesting:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/.*"
              - method: "POST"
                path: "/api/v1/orders"

Look at the rules.http block. Cilium enforces L7 policies here, so you can allow specific HTTP methods and paths and reject the rest, no service mesh required. A compromised frontend can hit GET /api/v1/... but it can’t suddenly start calling some admin endpoint you never meant to expose. That’s blast radius reduction you can actually read.

DNS-Based Policies

Allow traffic to external services by DNS name:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-external-api
spec:
  endpointSelector:
    matchLabels:
      app: backend
  egress:
    - toFQDNs:
        - matchName: "api.stripe.com"
        - matchPattern: "*.amazonaws.com"
      toPorts:
        - ports:
            - port: "443"

Cilium intercepts the DNS queries and updates the policy when the IPs behind those names change, so you write the rule against a name a human understands instead of a CIDR that rotates out from under you.

Cluster-Wide Policies

Apply policies across all namespaces:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-external-by-default
spec:
  endpointSelector: {}
  egress:
    - toEntities:
        - cluster
        - host
    - toFQDNs:
        - matchPattern: "*.internal.company.com"
  egressDeny:
    - toEntities:
        - world

This denies internet access by default and only lets internal traffic through unless you explicitly open something. Default-deny egress is one of those things that feels paranoid until the day a dependency you forgot about tries to phone home and you catch it in the logs.

Hubble: Network Observability

Remember the iptables evening I opened with? Hubble is the answer to it. It’s Cilium’s observability layer, and it shows you what your network is actually doing instead of leaving you to infer it from rules.

# Enable Hubble
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
  metrics:
    enabled:
      - dns
      - drop
      - tcp
      - flow
      - icmp
      - http

Hubble CLI

# Install Hubble CLI
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin

# Port-forward to Hubble Relay
cilium hubble port-forward &

# Observe flows
hubble observe --namespace production

# Filter by pod
hubble observe --pod production/api-xyz123

# Filter by verdict
hubble observe --verdict DROPPED

# Filter by HTTP
hubble observe --protocol http --http-status 500

Hubble UI

Access the visual network flow map:

cilium hubble ui
# Opens browser to http://localhost:12000

The UI shows:

  • Service dependency graph
  • Real-time traffic flows
  • Policy verdicts (allowed/dropped)
  • HTTP request/response details

Hubble Metrics

Export to Prometheus:

hubble:
  metrics:
    serviceMonitor:
      enabled: true
    enabled:
      - dns:query;ignoreAAAA
      - drop
      - tcp
      - flow
      - icmp
      - http

Useful metrics:

  • hubble_flows_processed_total, total observed flows
  • hubble_drop_total, dropped packets by reason
  • hubble_http_requests_total, HTTP requests by method and status
  • hubble_dns_queries_total, DNS queries by type and response

Service Mesh (Without Sidecars)

This is the advanced layer, and the one where Cilium quietly replaces a whole category of tooling. You can get service mesh features without running a sidecar proxy next to every pod:

# Enable L7 proxy
l7Proxy: true

# Enable mutual TLS
authentication:
  mutual:
    spiffe:
      enabled: true
      trustDomain: cluster.local

Features available:

  • mTLS between services (using SPIFFE identities)
  • L7 load balancing with retries
  • Traffic management (canary, header-based routing)
  • Observability (L7 metrics, distributed tracing)

The thing I care about: no sidecar containers. The eBPF dataplane handles all of it, which cuts the resource overhead and the latency you’d otherwise pay for an Envoy next to every pod. Fewer moving parts to reason about when something breaks at 2am.

apiVersion: cilium.io/v2
kind: CiliumEnvoyConfig
metadata:
  name: api-routing
spec:
  services:
    - name: api
      namespace: production
  backendServices:
    - name: api-v1
      namespace: production
    - name: api-v2
      namespace: production
  resources:
    - "@type": type.googleapis.com/envoy.config.listener.v3.Listener
      # ... Envoy configuration for traffic splitting

BGP for Bare Metal

If you self-host on real hardware like I do, you don’t get a cloud load balancer handed to you. Cilium speaks BGP, so your own network gear can route to pods directly:

# Enable BGP
bgpControlPlane:
  enabled: true

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: rack-bgp
spec:
  nodeSelector:
    matchLabels:
      rack: rack-1
  virtualRouters:
    - localASN: 65001
      exportPodCIDR: true
      neighbors:
        - peerAddress: "10.0.0.1/32"
          peerASN: 65000

Your pods get routable IPs, advertised via BGP to your network infrastructure.

Bandwidth Management

Cilium can enforce bandwidth limits:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: "10M"
    kubernetes.io/egress-bandwidth: "10M"
spec:
  containers:
    - name: app
      image: my-app:latest

This runs in eBPF rather than as tc rules, so it’s more efficient and the limits behave predictably under load.

Encryption

Encrypt every pod-to-pod flow with one block of config:

# WireGuard encryption (recommended)
encryption:
  enabled: true
  type: wireguard

# Or IPsec
encryption:
  enabled: true
  type: ipsec

I run WireGuard. It’s faster and simpler, and the kernel module is already there. Reach for IPsec only if you have a compliance certification that specifically demands it.

Verify encryption:

kubectl -n kube-system exec ds/cilium -- cilium status | grep Encryption
# Encryption: Wireguard [NodeEncryption: Disabled, cilium_wg0 (Pubkey: xxx, Port: 51871, Peers: 2)]

Multi-Cluster with Cluster Mesh

Connect multiple Cilium clusters:

# Enable Cluster Mesh on each cluster
cilium clustermesh enable

# Connect clusters
cilium clustermesh connect --destination-context cluster2

Features:

  • Global services, the same service reachable from any cluster
  • Cross-cluster network policies, so you can allow traffic from cluster1 to cluster2 pods
  • Shared identities, giving you consistent security policies across clusters
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-from-cluster1
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.cilium.k8s.policy.cluster: cluster1
            app: frontend

Troubleshooting

Check Endpoint Status

kubectl -n kube-system exec ds/cilium -- cilium endpoint list

Check Policy Verdicts

kubectl -n kube-system exec ds/cilium -- cilium policy get

Debug Dropped Traffic

hubble observe --verdict DROPPED

Check BPF Maps

kubectl -n kube-system exec ds/cilium -- cilium bpf lb list
kubectl -n kube-system exec ds/cilium -- cilium bpf endpoint list

My Production Configuration

Here’s the full picture, the actual config I run. Compare it to the bare cilium install we started with and you can see how far up the stack we climbed:

cilium:
  kubeProxyReplacement: true
  k8sServiceHost: "10.0.0.1"
  k8sServicePort: "6443"

  ipam:
    mode: kubernetes

  # Enable Hubble observability
  hubble:
    enabled: true
    relay:
      enabled: true
    ui:
      enabled: true
    metrics:
      enabled:
        - dns:query
        - drop
        - tcp
        - flow
        - http

  # WireGuard encryption
  encryption:
    enabled: true
    type: wireguard

  # L7 policies
  l7Proxy: true

  # Operator HA
  operator:
    replicas: 2

  # Resource limits
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi

Why each piece is there:

  • kube-proxy replacement, because it’s simpler and faster, and it removes the layer I couldn’t read
  • WireGuard encryption, because pod-to-pod traffic should be encrypted and this costs me almost nothing to turn on
  • Hubble metrics, exported to Prometheus so I get alerts instead of surprises
  • L7 proxy, for HTTP-aware policies without dragging in a separate mesh

Why Cilium?

To be fair, the traditional CNI plugins work. Calico, Flannel, Weave, they all route packets and plenty of clusters run on them happily. My problem with them is what they’re built on: iptables, a tool from 1998 that I end up reading by hand when things go sideways.

Cilium lines up with how I want to run infrastructure. The dataplane is programmable eBPF in the kernel. Policies are identity-based, so they describe what a workload is rather than which IP it happens to have today. The traffic is observable through Hubble, so I see flows instead of inferring them from rule order. And service mesh, BGP, and encryption all come from one tool instead of four.

That last point matters more than it looks. As I argued in Sovereign Infrastructure, I want to understand what I’m running, and a black box in the network path is the worst kind of splinter. Cilium gives me the visibility iptables never could, catches L7 attacks that L3/L4 rules miss, and runs faster while doing it. The fact that I can read what it’s doing is the whole reason I switched.

iptables did its job for decades. But I want a network layer I can inspect, not one I restart and pray over. eBPF gives me that, and Cilium is how I put it to work.