Deploying High-Availability Kubernetes on Bare Metal & VPS with K3s and Cilium eBPF

1. Architectural Motivation: Why K3s + Cilium eBPF on Bare Metal or VPS?

Managed Kubernetes control planes (AWS EKS, GCP GKE, Azure AKS) simplify cluster lifecycle management, but incur per-cluster management fees ($73/month per cluster), expensive managed NAT Gateways, and cross-AZ data transfer charges ($0.02/GB round-trip). For engineering teams running stateful workloads or predictable stateless services on bare metal or KVM VPS nodes, combining K3s with Cilium eBPF provides a CNCF-conformant Kubernetes stack with minimal memory overhead and kernel-native networking.

Why Replace kube-proxy and iptables with Cilium eBPF?

In a traditional Kubernetes cluster, kube-proxy translates every Kubernetes Service (ClusterIP, NodePort) into chains of iptables (or ipvs) rules. As a cluster scales to hundreds or thousands of Services: - Packet forwarding latency degrades because iptables evaluates rules sequentially ($O(N)$ complexity). - Updating rule tables during frequent Pod autoscaling causes lock contention and transient latency spikes. - NetworkPolicy enforcement requires additional user-space or netfilter hooks.

When Cilium runs with kubeProxyReplacement=true, it attaches eBPF programs directly to Linux socket operations (sockops) and network device traffic control (tc/XDP) hooks. Service load-balancing uses in-kernel eBPF hash maps ($O(1)$ lookup time), bypassing intermediate netfilter chains entirely.

2. Prerequisites & Host Kernel Preparation

Before bootstrapping your cluster on three Linux servers (for example, three 4-vCPU / 8 GB RAM nodes connected over a private VPC VLAN 10.10.0.11, 10.10.0.12, and 10.10.0.13):

1. Odd Number of Control-Plane Nodes (Quorum Rule): Raft consensus in etcd requires a strict majority ($\lfloor N/2
floor + 1$) of nodes to agree on every state transition. A 3-node cluster tolerates 1 node failure; a 2-node cluster has *lower* availability than a single node because losing either host breaks quorum.
2. Low-Latency Local NVMe for etcd: Verify that disk fsync latency on /var/lib/rancher/k3s/server/db/etcd is consistently below 10ms (ideally < 2ms).
3. Linux Kernel Sysctl Tuning: Configure kernel networking parameters across all nodes before installing K3s:

cat << 'EOF' | sudo tee /etc/sysctl.d/99-kubernetes-cilium.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 16384
EOF
sudo sysctl --system

3. Bootstrapping the 3-Node K3s Control Plane (Without Default CNI or Kube-Proxy)

To hand full datapath control over to Cilium, we must instruct K3s to disable its built-in flannel CNI, disable kube-proxy, disable the default network policy controller, and bind node communication to the private network interface:

# On Node 1 (10.10.0.11) — Initialize the Embedded etcd Cluster
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server \
  --cluster-init \
  --flannel-backend=none \
  --disable-kube-proxy \
  --disable-network-policy \
  --disable=traefik \
  --node-ip=10.10.0.11 \
  --advertise-address=10.10.0.11 \
  --tls-san=10.10.0.11 \
  --tls-san=k8s-api.internal.example.com \
  --write-kubeconfig-mode=0640" sh -
# Retrieve the cluster join token from Node 1
sudo cat /var/lib/rancher/k3s/server/node-token

Next, join Node 2 (10.10.0.12) and Node 3 (10.10.0.13) to form the 3-node HA control plane:

# On Node 2 (10.10.0.12) — Repeat with 10.10.0.13 on Node 3
curl -sfL https://get.k3s.io | K3S_TOKEN="<PASTE_NODE_TOKEN_HERE>" INSTALL_K3S_EXEC="server \
  --server https://10.10.0.11:6443 \
  --flannel-backend=none \
  --disable-kube-proxy \
  --disable-network-policy \
  --disable=traefik \
  --node-ip=10.10.0.12 \
  --advertise-address=10.10.0.12 \
  --tls-san=k8s-api.internal.example.com" sh -

*Note:* At this stage, kubectl get nodes will show all three nodes in NotReady state until Cilium CNI is installed.

4. Installing Cilium eBPF with WireGuard Pod-to-Pod Encryption

Install the official Cilium CLI on Node 1 and deploy Cilium pointing directly at the local K3s API server (127.0.0.1:6443 or your internal HA virtual IP):

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
# Deploy Cilium with eBPF kube-proxy replacement, Hubble observability, and WireGuard encryption
cilium install \
  --version 1.16.3 \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=10.10.0.11 \
  --set k8sServicePort=6443 \
  --set ipam.operator.clusterPoolIPv4PodCIDRList="{10.42.0.0/16}" \
  --set bpf.masquerade=true \
  --set encryption.enabled=true \
  --set encryption.type=wireguard \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true
# Wait for all Cilium pods to become Ready and verify eBPF kube-proxy replacement status
cilium status --wait
kubectl exec -n kube-system ds/cilium -- cilium-dbg status | grep KubeProxyReplacement

Verify that KubeProxyReplacement reports True and that Encryption reports Wireguard [cilium_wg0]. With encryption.type=wireguard enabled, all cross-node pod traffic is transparently encrypted inside the Linux kernel using ephemeral WireGuard keys managed automatically by the Cilium agent.

5. Automated Encrypted etcd Snapshots to S3 / Cloudflare R2

Even with 3-node Raft replication, accidental namespace deletion or corrupted CRD upgrades require off-cluster backups. Configure K3s to push automated etcd snapshots to an S3-compatible bucket (such as AWS S3 or zero-egress Cloudflare R2) every 6 hours:

# /etc/rancher/k3s/config.yaml
etcd-s3: true
etcd-s3-endpoint: "<ACCOUNT_ID>.r2.cloudflarestorage.com"
etcd-s3-bucket: "prod-k3s-etcd-backups"
etcd-s3-access-key: "<R2_ACCESS_KEY>"
etcd-s3-secret-key: "<R2_SECRET_KEY>"
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 28

Test your restore procedure regularly in staging before relying on snapshots in production.

6. Zero-Downtime Automated K3s Upgrades (system-upgrade-controller) & Host Kernel Tuning

Running production Kubernetes on bare metal or KVM VPS requires tuning host sysctl limits so high pod density does not exhaust Linux inotify watches or conntrack tables, and automating rolling cluster upgrades safely.

Mandatory Host Kernel /etc/sysctl.d/90-k3s-production.conf Baseline

Apply the following kernel parameters on every K3s control-plane and worker node prior to joining the cluster:

# /etc/sysctl.d/90-k3s-production.conf
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 1048576
fs.file-max = 2097152
net.core.somaxconn = 32768
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
vm.overcommit_memory = 1
vm.panic_on_oom = 0
vm.swappiness = 0

Verifying Cilium eBPF Datapath & Etcd Raft Quorum Health

Before draining any control-plane node for kernel patching or K3s version upgrades, verify that all 3 embedded etcd members are healthy and Cilium BPF maps show zero pressure:

# 1. Verify Cilium eBPF kube-proxy replacement and WireGuard encryption state
cilium status --verbose
cilium bpf Map list
# 2. Check embedded etcd cluster member health and Raft leader latency
sudo k3s kubectl get nodes -o wide
sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
  endpoint health --cluster -w table

For deep-dive packet observability on top of this stack, see our companion guide on Cilium Hubble & Prometheus Kubernetes Network Observability.