Cilium: eBPF Networking for Kubernetes

TL;DR Cilium is the CNI I run in every cluster. It replaces kube-proxy and its iptables chains with eBPF programs in the kernel, enforces network policies on identity and on HTTP paths, and Hubble shows me every flow and every drop. WireGuard between the pods, BGP to the switch, no sidecars. My production values are at the bottom.

The first time a service stopped resolving in one of my clusters, I spent an evening reading iptables chains. Hundreds of rules, generated by kube-proxy, evaluated top to bottom. I never found the problem. I restarted the node and it went away.

That bothered me more than the outage did. I was running something I couldn’t read.

It’s why I moved to Cilium. It uses eBPF to push the networking logic into the Linux kernel and skips iptables entirely. It’s faster, you can see what your traffic does, and a network policy stops being a guess.

This is what I run in my clusters now. I’ll start with the smallest piece that works and add to it, so you can stop wherever you have enough.

What eBPF is

Start here, because the rest only makes sense once this clicks. eBPF, extended Berkeley Packet Filter, lets you run sandboxed programs inside the Linux kernel without patching the kernel source or loading a module.

Traditional networking leans on iptables, a chain of rules walked one by one until something matches. An eBPF program gets compiled and runs in the kernel, and it looks things up in maps instead of walking a list. The difference shows up when you scale it.

Iptables with 10,000 services means a linear scan that gets slower with every service you add. Cilium with 10,000 services does one constant-time map lookup. Same workload, very different behaviour under load. And when something is off, I can dump the maps instead of guessing which of a few hundred rules misfired.

Installing Cilium

The smallest version that works. Swap your existing CNI for Cilium and check that it came up.

# Install Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz

# Install Cilium in cluster
cilium install --version 1.15.0

# Verify installation
cilium status --wait

The CLI is fine for a first look, but I don’t run anything by hand twice. Here’s the same install as an ArgoCD Application, so the network layer lives in Git like everything else in the cluster:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cilium
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://helm.cilium.io/
    chart: cilium
    targetRevision: 1.15.0
    helm:
      values: |
        kubeProxyReplacement: true
        k8sServiceHost: "10.0.0.1"
        k8sServicePort: "6443"

        hubble:
          enabled: true
          relay:
            enabled: true
          ui:
            enabled: true

        operator:
          replicas: 2

        ipam:
          mode: kubernetes

  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system

kube-proxy replacement

It’s the first thing I turn on after the install. Cilium can replace kube-proxy entirely and implement Kubernetes Services in eBPF. It’s how I got rid of the iptables chains this post started with:

# Cilium values
kubeProxyReplacement: true
k8sServiceHost: "10.0.0.1"  # API server IP
k8sServicePort: "6443"

What you get:

• No iptables rules for services
• Direct server return, which is faster
• Socket-level load balancing, where the decision happens once at connect() instead of on every packet
• Maglev hashing, so backend selection stays consistent

Check that it’s on:

kubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxyReplacement
# KubeProxyReplacement: True [eth0 10.0.0.10 (Direct Routing)]

Network policies

Now the part that changes how you think about security. Cilium speaks the standard Kubernetes NetworkPolicy, and it adds its own CiliumNetworkPolicy on top with a lot more reach. If you’ve never written a policy, start with the standard one.

Kubernetes NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

CiliumNetworkPolicy, the extended one

Once that makes sense, the native policy is where it gets interesting:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/.*"
              - method: "POST"
                path: "/api/v1/orders"

Look at the rules.http block. Cilium enforces L7 policy there, so you allow specific HTTP methods and paths and reject the rest, without a service mesh. A compromised frontend can still call GET /api/v1/.... It can’t start calling an admin endpoint you never meant to expose. I can read that policy and tell you what a breached pod reaches, which I couldn’t with iptables.

DNS-based policies

Allow traffic to an external service by name:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-external-api
spec:
  endpointSelector:
    matchLabels:
      app: backend
  egress:
    - toFQDNs:
        - matchName: "api.stripe.com"
        - matchPattern: "*.amazonaws.com"
      toPorts:
        - ports:
            - port: "443"

Cilium intercepts the DNS queries and updates the policy when the IPs behind those names change. You write the rule against a name a human understands instead of a CIDR that rotates out from under you.

Cluster-wide policies

The same thing across all namespaces:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-external-by-default
spec:
  endpointSelector: {}
  egress:
    - toEntities:
        - cluster
        - host
    - toFQDNs:
        - matchPattern: "*.internal.company.com"
  egressDeny:
    - toEntities:
        - world

This denies internet access by default and lets only internal traffic through, unless you open something on purpose. Default-deny egress feels paranoid right up to the day a dependency you forgot about tries to phone home and you catch it in the logs.

Hubble, the observability layer

Remember the iptables evening at the top? Hubble is what I wanted that evening. It’s Cilium’s observability layer, and it shows what the network is doing instead of leaving you to infer it from rule order.

# Enable Hubble
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
  metrics:
    enabled:
      - dns
      - drop
      - tcp
      - flow
      - icmp
      - http

Hubble CLI

# Install Hubble CLI
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin

# Port-forward to Hubble Relay
cilium hubble port-forward &

# Observe flows
hubble observe --namespace production

# Filter by pod
hubble observe --pod production/api-xyz123

# Filter by verdict
hubble observe --verdict DROPPED

# Filter by HTTP
hubble observe --protocol http --http-status 500

Hubble UI

Here’s the visual flow map:

cilium hubble ui
# Opens browser to http://localhost:12000

The UI shows:

• the service dependency graph
• traffic flows, live
• policy verdicts, allowed or dropped
• HTTP request and response details

Hubble metrics

Export to Prometheus:

hubble:
  metrics:
    serviceMonitor:
      enabled: true
    enabled:
      - dns:query;ignoreAAAA
      - drop
      - tcp
      - flow
      - icmp
      - http

Useful ones:

• hubble_flows_processed_total, total observed flows
• hubble_drop_total, dropped packets by reason
• hubble_http_requests_total, HTTP requests by method and status
• hubble_dns_queries_total, DNS queries by type and response

Service mesh without sidecars

Here Cilium replaces a whole category of tooling. You get service mesh features without a sidecar proxy next to every pod:

# Enable L7 proxy
l7Proxy: true

# Enable mutual TLS
authentication:
  mutual:
    spiffe:
      enabled: true
      trustDomain: cluster.local

What that gives you:

• mTLS between services, on SPIFFE identities
• L7 load balancing with retries
• Traffic management, canary and header-based routing
• Observability, L7 metrics and distributed tracing

The part I care about is the missing sidecar. The eBPF dataplane handles all of it, which saves the memory and the latency you’d otherwise pay for an Envoy next to every pod. And it’s one fewer container to look at when a request goes missing.

apiVersion: cilium.io/v2
kind: CiliumEnvoyConfig
metadata:
  name: api-routing
spec:
  services:
    - name: api
      namespace: production
  backendServices:
    - name: api-v1
      namespace: production
    - name: api-v2
      namespace: production
  resources:
    - "@type": type.googleapis.com/envoy.config.listener.v3.Listener
      # ... Envoy configuration for traffic splitting

BGP for bare metal

If you self-host on real hardware like I do, there’s no cloud handing you a load balancer. Cilium speaks BGP, so your own switch can route to pods directly:

# Enable BGP
bgpControlPlane:
  enabled: true

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: rack-bgp
spec:
  nodeSelector:
    matchLabels:
      rack: rack-1
  virtualRouters:
    - localASN: 65001
      exportPodCIDR: true
      neighbors:
        - peerAddress: "10.0.0.1/32"
          peerASN: 65000

Your pods get routable IPs, advertised over BGP to your network gear.

Bandwidth limits

Cilium can cap bandwidth per pod:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: "10M"
    kubernetes.io/egress-bandwidth: "10M"
spec:
  containers:
    - name: app
      image: my-app:latest

This runs in eBPF instead of as tc rules, so it’s cheaper and the limit holds under load.

Encryption

Encrypt every pod-to-pod flow with one block of config:

# WireGuard encryption (recommended)
encryption:
  enabled: true
  type: wireguard

# Or IPsec
encryption:
  enabled: true
  type: ipsec

I run WireGuard. It’s faster and simpler, and the kernel module is already there. IPsec only if a compliance certification demands it by name.

Check it:

kubectl -n kube-system exec ds/cilium -- cilium status | grep Encryption
# Encryption: Wireguard [NodeEncryption: Disabled, cilium_wg0 (Pubkey: xxx, Port: 51871, Peers: 2)]

Multi-cluster with Cluster Mesh

Connect several Cilium clusters:

# Enable Cluster Mesh on each cluster
cilium clustermesh enable

# Connect clusters
cilium clustermesh connect --destination-context cluster2

What that gives you:

• Global services, the same service reachable from any cluster
• Cross-cluster network policies, so you can allow traffic from cluster1 into cluster2 pods
• Shared identities, so one policy means the same thing in every cluster

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-from-cluster1
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.cilium.k8s.policy.cluster: cluster1
            app: frontend

Troubleshooting

Endpoint status

kubectl -n kube-system exec ds/cilium -- cilium endpoint list

Policy verdicts

kubectl -n kube-system exec ds/cilium -- cilium policy get

Dropped traffic

hubble observe --verdict DROPPED

BPF maps

kubectl -n kube-system exec ds/cilium -- cilium bpf lb list
kubectl -n kube-system exec ds/cilium -- cilium bpf endpoint list

What I run in production

Here’s the whole config, the values I run. Next to the bare cilium install at the top, this is everything the post turned on:

cilium:
  kubeProxyReplacement: true
  k8sServiceHost: "10.0.0.1"
  k8sServicePort: "6443"

  ipam:
    mode: kubernetes

  # Enable Hubble observability
  hubble:
    enabled: true
    relay:
      enabled: true
    ui:
      enabled: true
    metrics:
      enabled:
        - dns:query
        - drop
        - tcp
        - flow
        - http

  # WireGuard encryption
  encryption:
    enabled: true
    type: wireguard

  # L7 policies
  l7Proxy: true

  # Operator HA
  operator:
    replicas: 2

  # Resource limits
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi

Why each piece is there:

• kube-proxy replacement, because it’s simpler and faster, and it removes the layer I couldn’t read
• WireGuard encryption, because pod-to-pod traffic should be encrypted and this costs me almost nothing
• Hubble metrics, exported to Prometheus so I get an alert instead of a surprise
• L7 proxy, for HTTP-aware policies without a separate mesh

Why Cilium

To be fair, the traditional CNI plugins work. Calico , Flannel , Weave, they all route packets and plenty of clusters run on them without complaint. My problem is what they’re built on. Iptables is a tool from 1998, and I end up reading it by hand when things go sideways.

Cilium fits how I want to run infrastructure. The dataplane is eBPF in the kernel, and I can dump its maps. Policies are identity-based, so they describe what a workload is rather than which IP it has today. Hubble shows me flows instead of leaving me to infer them from rule order. And service mesh, BGP and encryption come from one tool instead of four.

As I wrote in Sovereign Infrastructure, I want to understand what I’m running, and the network path is the worst place for a black box. Cilium catches L7 attacks that L3/L4 rules miss and runs faster while doing it. But the reason I switched is the outage I opened with. The next time a service stops resolving, I run hubble observe --verdict DROPPED and read the answer, instead of restarting a node and hoping.

• Networking
• Zero Trust