eBPF and Cilium FAQ

What is eBPF in Kubernetes?

eBPF lets you run sandboxed programs inside the Linux kernel without changing kernel source or loading modules. In Kubernetes it is used to handle networking, observability and security at the kernel level, replacing slower userspace paths such as iptables rule chains.

Is Cilium better than kube-proxy?

For large clusters, generally yes. kube-proxy in iptables mode evaluates rule chains sequentially, so per-packet cost grows with Service count. Cilium uses eBPF hash-table lookups whose cost stays broadly constant. On a small cluster the difference is not worth a migration.

What kernel version does Cilium require?

Cilium runs on 4.19 and later, but 5.10 or newer is recommended for the full feature set including the bandwidth manager, egress gateway and better XDP support. Full kube-proxy replacement wants 5.4 or later.

What is Hubble used for?

Hubble is Cilium's observability layer. It provides flow-level visibility into what is talking to what, why connections are dropped, and how services relate — without sidecars or application instrumentation. 'hubble observe --verdict DROPPED' is the fastest way to diagnose a policy problem.

Does eBPF improve Kubernetes performance?

It removes the per-packet cost of long iptables chains and eliminates sidecar proxy hops in a sidecarless mesh. The gain scales with cluster size — meaningful on clusters with thousands of Services, largely invisible on a handful.

Can Cilium replace my service mesh?

For L3/L4 policy, load balancing and observability, often yes, and without a sidecar per pod. Cilium also handles mutual authentication and L7 policy. Advanced traffic management such as fine-grained canary splitting may still need a dedicated mesh.

Troubleshooting Cilium

Pods stuck in ContainerCreating after install

Nearly always CNI configuration. Check the agent logs and confirm the config was written:

kubectl -n kube-system logs ds/cilium --tail=50
ls /etc/cni/net.d/          # expect a Cilium conflist

If a previous CNI's config file is still present and sorts earlier alphabetically, the kubelet will keep using it. Remove the stale file and restart the kubelet.

Connectivity test fails but status is green

cilium status reports agent health, not data-path correctness. A failing connectivity test with a healthy status usually points at kube-proxy replacement being partially configured, or at a cloud firewall blocking the VXLAN or Geneve tunnel port between nodes.

Traffic dropped after applying a policy

hubble observe --verdict DROPPED --last 50
kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list

The endpoint list shows the enforcement state per endpoint. Combined with the drop reason from Hubble, that is normally enough to identify the missing rule in one pass.

High memory use on the agent

Usually eBPF map sizing on large clusters. The maps are pre-allocated, so a cluster with many endpoints needs explicit tuning rather than defaults. Size bpf.mapDynamicSizeRatio or the individual map limits against your endpoint count.

Network Policy Beyond L3/L4

Standard Kubernetes NetworkPolicy stops at IP and port. CiliumNetworkPolicy extends to the application layer, which is where the useful restrictions live. This one allows GETs to a specific path and denies everything else — without a sidecar proxy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-checkout-to-payments
spec:
  endpointSelector:
    matchLabels:
      app: payments
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: checkout
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: "/api/v1/charge"

Note fromEndpoints with label selectors rather than CIDR blocks. The policy follows the workload, not its address.

One warning worth taking seriously: the moment any policy selects an endpoint, that endpoint defaults to deny for everything not explicitly allowed. Roll policies out in audit mode first and watch Hubble for drops before enforcing.

Using Hubble to Actually Debug Something

Hubble's value is answering "why did this connection fail" without adding sidecars or instrumenting the application. The commands below are the ones worth memorising.

# Live flow feed across the cluster
hubble observe --follow

# Only what a specific pod is doing
hubble observe --pod default/checkout --follow

# The question you usually have: what is being dropped, and why?
hubble observe --verdict DROPPED --last 100

# Narrow to one namespace and port
hubble observe --namespace payments --port 5432

# HTTP-level visibility, when an L7 policy is in force
hubble observe --protocol http --verdict DROPPED

--verdict DROPPED is the one that earns its keep. Hubble reports the reason alongside the flow — "Policy denied", for instance — which turns a vague connectivity complaint into a specific policy line to fix.

For a visual service map, port-forward the UI:

cilium hubble ui

Replacing kube-proxy: What Actually Changes

This is the change people mean when they say Cilium is faster, and it is worth understanding rather than taking on faith.

kube-proxy in iptables mode writes a rule chain for every Service and endpoint. Rule evaluation is essentially sequential, so the cost of matching a packet grows with the size of the chain. A cluster with a handful of Services never notices. A cluster with thousands does — both in per-packet latency and in the time taken to reprogram rules when endpoints change.

Cilium replaces that with eBPF programs attached to kernel hooks, using hash-table lookups whose cost stays flat as Service count grows.

Confirm the replacement is actually active — a partial configuration silently leaves kube-proxy handling some paths:

kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement

In older Cilium versions the in-pod binary is cilium rather than cilium-dbg. If the command above is not found, try it without the suffix.

The identity model is the part that changes how you work. Cilium assigns an identity derived from workload labels, so a policy keeps applying as pods are rescheduled and IPs churn. Anyone who has debugged an IP-based policy after a rolling update will appreciate why that matters.

Installing Cilium on Kubernetes

The CLI is the fastest route for a test cluster:

# Install the Cilium CLI, then:
cilium install

# Wait for the control plane to settle
cilium status --wait

# Verify data-path connectivity end to end.
# This spins up test workloads and takes several minutes.
cilium connectivity test

For anything you intend to keep, use Helm so the configuration is reviewable and reproducible:

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<API_SERVER_HOST> \
  --set k8sServicePort=<API_SERVER_PORT> \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Version note: kubeProxyReplacement took the values strict, partial or disabled in older Cilium releases and true/false in current ones. Check the values reference for the chart version you are installing — this flag is a frequent source of failed upgrades.

cilium connectivity test is worth the wait. It exercises pod-to-pod, pod-to-service and egress paths, and it catches misconfiguration that a green cilium status will happily hide.

Requirements Before You Start

eBPF is a kernel technology, so the kernel version is the constraint that matters most. Check it before anything else:

uname -r          # on every node, not just one

Cilium runs on older kernels, but the interesting features are gated behind newer ones. As a rough guide:

Managed Kubernetes changes the picture. On EKS, GKE and AKS the node image dictates your kernel, and each provider has its own Cilium install path — particularly on EKS, where you choose between replacing the AWS VPC CNI and chaining onto it. Always check the Cilium documentation for your specific platform before running a generic install.