Kubernetes eBPF Network Observability: Cilium Hubble, Service Dependency Graphs & DNS Debugging

1. Why Traditional tcpdump and Sidecar Proxies Struggle at Cloud-Native Scale

When a production Kubernetes microservice experiences intermittent HTTP 502/504 gateway timeouts or ETIMEDOUT connection resets, diagnosing the root cause with traditional tools is notoriously difficult:
- Ephemeral container IP addresses (10.42.x.y) are recycled every time a Deployment scales or rolls out, rendering raw IP flow logs (VPC Flow Logs) meaningless without Kubernetes Pod/Service metadata correlation.
- Deploying Envoy sidecar proxies inside every single Pod (traditional sidecar service mesh) adds 50–150 MB of RAM per Pod and doubles the number of TCP socket traversals.

Cilium Hubble solves this by tapping directly into the in-kernel eBPF datapath. Because Cilium already knows the Kubernetes identity (namespace, pod, labels, service) of every endpoint inside kernel eBPF maps, Hubble exports zero-sidecar L3/L4 and L7 (DNS, HTTP, Kafka, gRPC) flow visibility with negligible overhead.

2. Real-Time Packet Drop & NetworkPolicy Debugging with the hubble CLI

The fastest way to answer *"Why can't Pod A connect to Service B?"* is using hubble observe filtered by verdict DROPPED:

# Port-forward to the cluster's Hubble Relay service
cilium hubble port-forward &
# 1. Watch all dropped packets across the cluster in real time with drop reason codes
hubble observe --verdict DROPPED --follow
# 2. Inspect L3/L4 and L7 flows specifically for the 'checkout-api' pod in 'prod' namespace
hubble observe --namespace prod --pod checkout-api --last 50
# 3. Inspect all DNS queries and NXDOMAIN / SERVFAIL responses across the cluster
hubble observe --protocol dns --follow

When a CiliumNetworkPolicy or Kubernetes NetworkPolicy blocks a connection, Hubble immediately prints Policy denied (L3-Only) along with the exact source and destination Kubernetes labels (k8s:app=checkout-api -> k8s:app=postgres-primary:5432), eliminating hours of guesswork.

3. Solving the Infamous Kubernetes ndots:5 CoreDNS Amplification Bug

One of the most frequent findings when inspecting hubble observe --protocol dns in a Kubernetes cluster is discovering that every single external HTTPS call (e.g., to api.stripe.com or s3.amazonaws.com) generates 4 wasted NXDOMAIN DNS queries before resolving!

Why ndots:5 Happens by Default

Inside every Kubernetes Pod, /etc/resolv.conf defaults to:

nameserver 10.43.0.10
search prod.svc.cluster.local svc.cluster.local cluster.local example.com
options ndots:5

Because ndots:5 states that any hostname containing fewer than 5 dots is treated as a *relative* name first, looking up api.stripe.com (which has only 2 dots) forces glibc/musl to query CoreDNS sequentially for: 1. api.stripe.com.prod.svc.cluster.local -> NXDOMAIN 2. api.stripe.com.svc.cluster.local -> NXDOMAIN 3. api.stripe.com.cluster.local -> NXDOMAIN 4. api.stripe.com.example.com -> NXDOMAIN 5. api.stripe.com. -> NOERROR (Finally resolves!)

Two Production Fixes to Cut CoreDNS Load by 75%

1. Append a Trailing Dot to External FQDNs in Application Configs (api.stripe.com.): Or lower ndots to 2 in your Pod's dnsConfig:

spec:
     dnsConfig:
       options:
         - name: ndots
           value: "2"
         - name: single-request-reopen

2. Deploy NodeLocal DNSCache DaemonSet: Runs a lightweight local DNS caching agent (169.254.20.10) on every Kubernetes node over TCP/UDP, eliminating UDP conntrack table races and serving cached DNS responses in <0.2ms without crossing the cluster network.

4. High-Cardinality Prometheus Metrics Governance in Cilium Hubble

Enabling every Hubble metric dimensions (source_ip, destination_ip, pod_name, http_path) on a 100+ node Kubernetes cluster can generate millions of active Prometheus time series, exhausting TSDB memory within hours.

Production Helm Values for Bounded-Cardinality Hubble Metrics

Configure Hubble to aggregate metrics at the workload (Deployment/StatefulSet) and namespace level rather than individual ephemeral pod names or raw IP addresses:

# cilium-observability-values.yaml
hubble:
  enabled: true
  relay:
    enabled: true
  metrics:
    enabled:
      - dns:query;ignoreAAAA
      - drop:sourceContext=workload-name|reserved-identity;destinationContext=workload-name|reserved-identity
      - tcp:sourceContext=namespace;destinationContext=namespace
      - flow:sourceContext=namespace;destinationContext=namespace
      - httpV2:exemplars=true;labelsContext=source_namespace,source_workload,destination_namespace,destination_workload,traffic_direction
    serviceMonitor:
      enabled: true

5. Golden Alerting Rules for eBPF Datapath & CoreDNS Health

Deploy the following Prometheus alerting thresholds to detect Kubernetes networking regressions before application SLAs degrade:

| Prometheus Metric Expression | Alert Threshold | Root Cause & Remediation |
| :--- | :--- | :--- |
| rate(hubble_drop_total{reason!="POLICY_DENIED"}[5m]) > 5 | > 5 pkt/sec for 5m | Unexpected datapath packet drops (INVALID_SOURCE_IP, CT_MAP_INSERTION_FAILED); inspect conntrack table utilization (cilium bpf ct list global). |
| rate(hubble_dns_responses_total{rcode="SERVFAIL"}[5m]) > 1 | > 1 req/sec for 3m | Upstream VPC DNS resolver throttling or CoreDNS pod saturation; enable NodeLocal DNSCache DaemonSet. |
| histogram_quantile(0.99, sum(rate(hubble_http_request_duration_seconds_bucket[5m])) by (le, destination_workload)) > 1.5 | P99 > 1.5s | Downstream service saturation or connection pool queueing; trace exact traceId via OpenTelemetry exemplars. |