Kubernetes Mastery — From Fundamentals to Advanced Operations

Prerequisites: Familiarity with Linux command line, basic networking (TCP/IP, DNS, HTTP), containers and Docker fundamentals (images, containers, Dockerfiles), YAML syntax, and a general understanding of distributed systems concepts.

Why Kubernetes — The Problem It Solves

Containers changed how we package software — a single image that runs the same way on a laptop and in production. But the moment you go from one container to dozens (or thousands), a new class of problems appears. Which server should run each container? What happens when a container crashes at 3 AM? How do services find each other as instances come and go? Kubernetes exists to answer these questions.

Before diving into architecture and YAML manifests, it is worth understanding why this system was built and what life looks like without it. That context makes every Kubernetes concept that follows feel like a solution to a problem you already recognize.

The Pain of Manual Container Orchestration

Imagine you run a web application with three services: an API, a background worker, and a Redis cache. You have four servers. Without an orchestrator, you are the orchestrator. Here is what a typical deployment script looks like when humans manage containers directly:

#!/bin/bash
# deploy.sh — manual container deployment across 4 servers

# 1. Decide which server has capacity (check manually or guess)
ssh server-02 "docker stats --no-stream" | grep -v "0.00%"

# 2. Pull the new image on each target server
for host in server-01 server-02 server-03; do
  ssh $host "docker pull registry.example.com/api:v2.4.1"
done

# 3. Stop old containers one at a time (hope nobody notices)
ssh server-01 "docker stop api-1 && docker rm api-1"
ssh server-01 "docker run -d --name api-1 -p 8080:8080 \
  -e DB_HOST=10.0.1.50 -e DB_PASSWORD=s3cret \
  registry.example.com/api:v2.4.1"

# 4. Repeat for server-02 and server-03...
# 5. Update the load balancer config manually
# 6. Pray that DNS propagation is fast enough
# 7. Set up a cron job to restart crashed containers (?)

This script has serious problems. The database password is hardcoded in plain text. There is no health check — if the new version crashes on startup, traffic still routes to it. Rolling back means re-running the script with the old image tag and hoping the state is clean. Scaling up means provisioning a new server, installing Docker, copying SSH keys, and updating the script. At 3 AM, when server-02 dies, nobody restarts those containers until a human wakes up.

The Five Hard Problems at Scale

The script above is not a strawman — it is how many teams actually operated in 2014–2016. The problems it exposes fall into five categories that every container orchestrator must solve.

mindmap
  root((Container
    Orchestration))
    Scheduling
      Bin-packing onto nodes
      Resource-aware placement
      Affinity and anti-affinity
    Self-Healing
      Restart crashed containers
      Replace unresponsive nodes
      Health check enforcement
    Service Discovery
      Stable DNS names
      Load balancing across replicas
      Dynamic IP handling
    Rolling Updates
      Zero-downtime deploys
      Automatic rollback
      Canary and blue-green
    Configuration
      Secret management
      Environment-specific config
      Hot-reload support
    Scaling
      Horizontal pod autoscaling
      Cluster autoscaling
      Scale to zero
    Storage
      Persistent volumes
      Dynamic provisioning
      Storage class abstraction
    

1. Scheduling

Given 50 containers and 10 servers, which container goes where? You need to consider CPU and memory availability, disk I/O requirements, data locality, and constraints like "don't put two replicas of the same service on the same host." Doing this by hand is error-prone. Doing it automatically, thousands of times per day, is a scheduling problem that requires a dedicated system.

2. Service Discovery and Load Balancing

Containers get new IP addresses every time they restart. If your API talks to a Redis instance at a hardcoded IP, that breaks the moment Redis is rescheduled to a different node. You need a dynamic registry that tracks where each service is running and routes traffic accordingly — without requiring code changes.

3. Self-Healing

Containers crash. Nodes go offline. Disks fill up. A production-grade system must detect these failures and recover automatically — restart the container, reschedule it to a healthy node, and stop sending traffic to instances that are not ready. The goal is to match a desired state continuously, not just at deployment time.

4. Rolling Updates and Rollbacks

Deploying a new version should not cause downtime. You need to bring up new containers, verify they are healthy, shift traffic, and drain old containers — all without dropping requests. When a deployment goes wrong, you need to revert to the previous version in seconds, not minutes.

5. Secret and Configuration Management

Database passwords, API keys, and TLS certificates cannot live in Docker images or shell scripts. You need a way to inject secrets at runtime, rotate them without redeploying, and ensure that only authorized services can access them.

From Google Borg to Kubernetes

Kubernetes did not appear from a vacuum. Google had been running containers at scale since the early 2000s — long before Docker existed. Their internal systems, Borg and its successor Omega, orchestrated billions of containers per week across Google's global infrastructure. Gmail, Search, YouTube, and Maps all run on Borg.

In 2014, Google open-sourced a system that captured the core design principles of Borg and Omega but was rebuilt from scratch for the broader community. Three key ideas carried over:

• Declarative configuration. You describe what you want (3 replicas of this service, 512 MiB of RAM each), not how to get there. The system continuously works to make reality match your declaration.
• API-driven everything. Every operation — deploying, scaling, inspecting — goes through a versioned REST API. There is no SSH, no special CLI magic. The kubectl command is just an API client.
• Reconciliation loops. Controllers constantly compare actual state against desired state and take corrective action. This is what makes the system self-healing: it is always converging, not just executing a one-time script.

In 2015, Google donated Kubernetes to the newly formed Cloud Native Computing Foundation (CNCF). It was the first project to graduate from CNCF, and it catalyzed an entire ecosystem — from container runtimes (containerd, CRI-O) to service meshes (Istio, Linkerd) to observability tools (Prometheus, Jaeger).

The Declarative Model: Tell It What, Not How

The most important mental shift when learning Kubernetes is moving from imperative commands to declarative specifications. Instead of scripting a sequence of steps, you write a document that describes your desired end state and hand it to the system.

Here is the contrast. The imperative approach you saw earlier — SSH into servers, run Docker commands, update load balancers — is a recipe of how to deploy. If any step fails, the system is in an unknown state. The declarative approach is fundamentally different:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:v2.4.1
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password

Apply this file with a single command — kubectl apply -f api-deployment.yaml — and Kubernetes handles everything: finding nodes with available resources, pulling the image, starting containers, running health checks, configuring rolling updates, and injecting the secret. If a container crashes, Kubernetes restarts it. If a node dies, Kubernetes reschedules the pods elsewhere. The YAML file becomes the single source of truth.

Before and After: A Side-by-Side Comparison

Kubernetes vs. the Alternatives

Kubernetes is not the only container orchestrator. Docker Swarm, HashiCorp Nomad, and Amazon ECS all solve overlapping problems. Choosing between them depends on your scale, team expertise, and operational requirements.

The API-Driven Architecture

Everything in Kubernetes flows through its API server. When you run kubectl apply, you are making an HTTP request to the API. When a controller restarts a crashed pod, it reads from and writes to the same API. When a CI/CD pipeline deploys a new version, it hits the same endpoint. This uniform interface is what makes Kubernetes so extensible.

# kubectl is just an API client — you can do the same with curl
kubectl get pods -o json

# Equivalent raw API call (with authentication)
curl -k https://<api-server>:6443/api/v1/namespaces/default/pods \
  --header "Authorization: Bearer $TOKEN"

# Watch for real-time changes (the same mechanism controllers use)
curl -k https://<api-server>:6443/api/v1/namespaces/default/pods?watch=true \
  --header "Authorization: Bearer $TOKEN"

This design means you can build your own tools, dashboards, and automation on top of Kubernetes without special access. Custom Resource Definitions (CRDs) let you extend the API with your own object types, and operators — custom controllers that watch those objects — let you encode domain-specific operational knowledge into the cluster itself. You can define a PostgresCluster resource and let an operator handle backups, failovers, and upgrades automatically.

The Reconciliation Loop: Why Declarative Wins

The power of the declarative model becomes clear when things go wrong. In an imperative system, a failed step leaves you in a partially applied state — some servers have the new version, others have the old one. Recovery means writing more imperative logic to detect and fix the inconsistency.

In Kubernetes, every controller runs a continuous loop: observe the current state, compare it to the desired state, and act to close the gap. This loop runs constantly — not just at deploy time.

flowchart LR
    A["Desired State\n(YAML in etcd)"] -->|compare| B{"Diff?"}
    B -->|No difference| C["Do nothing\n(system is healthy)"]
    B -->|Drift detected| D["Take action\n(create, update, delete)"]
    D --> E["Actual State\n(running containers)"]
    E -->|observe| B
    C -->|wait, re-check| B

    style A fill:#4a9eff,color:#fff,stroke:#3380cc
    style B fill:#f5a623,color:#fff,stroke:#c4851c
    style D fill:#e74c3c,color:#fff,stroke:#b83a2e
    style E fill:#2ecc71,color:#fff,stroke:#25a25a
    style C fill:#2ecc71,color:#fff,stroke:#25a25a
    

Suppose you declare 3 replicas of your API. A node crashes, taking one replica with it. The Deployment controller notices the actual count (2) does not match the desired count (3), and it schedules a new pod on a healthy node. No human intervention. No pager alert. The system converges back to the desired state automatically. This is the fundamental difference between running containers and orchestrating them.

What You Will Build on This Foundation

This section answered the "why." You now understand the five core problems — scheduling, service discovery, self-healing, rolling updates, and configuration management — and why a declarative, API-driven system is the right abstraction for solving them at scale. You have seen how Kubernetes inherits battle-tested ideas from a decade of Google's internal infrastructure and where it fits relative to alternatives.

In the next section, Cluster Architecture at a Glance, you will see how Kubernetes is structured — the control plane components that make decisions and the node components that execute them. Every concept from this section — the API server, the reconciliation loop, the scheduler — maps directly to a specific component in the architecture.

Cluster Architecture at a Glance

A Kubernetes cluster is a set of machines — physical or virtual — organized into two distinct roles: control plane nodes that make decisions about the cluster, and worker nodes that run your application workloads. Every interaction you have with Kubernetes, from deploying an app to scaling a service, flows through this architecture.

Understanding this split is foundational. The control plane is the brain; worker nodes are the muscle. They communicate over well-defined APIs, and every component has a single, clear responsibility. This section maps out every piece so you know exactly what happens when you run kubectl apply.

The Two Halves of a Cluster

A Kubernetes cluster divides cleanly into the control plane and the data plane (worker nodes). The control plane manages cluster state — it decides what should run and where. Worker nodes execute those decisions — they actually run your containers. Here is what lives on each side:

Cluster Architecture Diagram

The diagram below shows how control plane and worker node components connect. Notice that every component communicates through the API server — it is the single hub. No component talks directly to etcd except the API server, and no component talks directly to another component's internal state.

graph TB
    subgraph CP["Control Plane Node(s)"]
        API["kube-apiserver"]
        ETCD["etcd"]
        SCHED["kube-scheduler"]
        CM["kube-controller-manager"]
        CCM["cloud-controller-manager"]
    end

    subgraph W1["Worker Node 1"]
        KL1["kubelet"]
        KP1["kube-proxy"]
        CR1["Container Runtime"]
        P1A["Pod A"]
        P1B["Pod B"]
    end

    subgraph W2["Worker Node 2"]
        KL2["kubelet"]
        KP2["kube-proxy"]
        CR2["Container Runtime"]
        P2A["Pod C"]
        P2B["Pod D"]
    end

    USER["👤 User / CI"]
    KUBECTL["kubectl"]

    USER --> KUBECTL
    KUBECTL -->|"HTTPS REST"| API
    API <-->|"read/write state"| ETCD
    SCHED -->|"watch & bind Pods"| API
    CM -->|"watch & reconcile"| API
    CCM -->|"cloud provider calls"| API

    KL1 -->|"watch assigned Pods"| API
    KL2 -->|"watch assigned Pods"| API
    KL1 --> CR1
    KL2 --> CR2
    CR1 --> P1A
    CR1 --> P1B
    CR2 --> P2A
    CR2 --> P2B
    KP1 -->|"watch Services/Endpoints"| API
    KP2 -->|"watch Services/Endpoints"| API
    

Desired State vs. Actual State

Kubernetes operates on a declarative model. You tell it what you want (desired state), and the system continuously works to make reality match. You never say "start 3 Pods" imperatively — you declare "there should be 3 replicas" and Kubernetes figures out the rest.

The desired state lives in etcd as resource specs (Deployments, Services, ConfigMaps). The actual state is the real-time status of nodes and containers as reported by kubelets. Controller loops in the controller manager are the bridge — they watch for drift between the two and take corrective action. If a Pod crashes, the ReplicaSet controller sees the replica count is below the desired number and tells the API server to create a new Pod.

Anatomy of a Request: From kubectl apply to Running Containers

Understanding the end-to-end request flow demystifies Kubernetes. Here is exactly what happens when you apply a Deployment manifest:

sequenceDiagram
    actor User
    participant kubectl
    participant API as kube-apiserver
    participant etcd
    participant CM as controller-manager
    participant Sched as kube-scheduler
    participant KL as kubelet
    participant CRT as Container Runtime

    User->>kubectl: kubectl apply -f deployment.yaml
    kubectl->>API: POST /apis/apps/v1/namespaces/default/deployments
    API->>API: Authenticate, Authorize (RBAC), Admission Control
    API->>etcd: Persist Deployment object
    etcd-->>API: Write confirmed

    Note over CM: Deployment controller watch triggers
    CM->>API: Read new Deployment, create ReplicaSet
    API->>etcd: Persist ReplicaSet object
    CM->>API: ReplicaSet controller creates Pod objects
    API->>etcd: Persist Pod objects (nodeName = "")

    Note over Sched: Scheduler watch triggers on unbound Pods
    Sched->>API: Read unbound Pods, evaluate node fitness
    Sched->>API: Bind Pod → selected Node (set nodeName)
    API->>etcd: Update Pod with nodeName

    Note over KL: Kubelet watch triggers for its node
    KL->>API: Read Pod spec assigned to this node
    KL->>CRT: Pull image, create & start containers
    CRT-->>KL: Containers running
    KL->>API: Report Pod status = Running
    API->>etcd: Update Pod status
    

The entire flow is asynchronous and event-driven. No component blocks waiting for the next one — they all use watches (long-lived HTTP connections to the API server) to react to changes. This is why Kubernetes can handle thousands of Pods across hundreds of nodes without a central orchestration bottleneck.

The Seven Stages in Plain Language

1. Submission. You run kubectl apply. kubectl reads your YAML, validates it client-side, and sends an HTTPS request to the API server.
2. Admission & Persistence. The API server authenticates your identity, checks RBAC authorization, runs admission webhooks (e.g., injecting sidecar containers), validates the schema, and writes the Deployment object to etcd.
3. Controller Reconciliation. The Deployment controller notices the new Deployment and creates a ReplicaSet. The ReplicaSet controller sees the new ReplicaSet and creates the specified number of Pod objects — but these Pods have no nodeName yet.
4. Scheduling. The scheduler watches for Pods without a node assignment. It evaluates each worker node against the Pod's resource requests, node selectors, affinity/anti-affinity rules, and taints/tolerations. It picks the best node and writes the binding back to the API server.
5. Kubelet Execution. The kubelet on the chosen node sees a new Pod assigned to it. It pulls the container image (if not cached), creates the container sandbox via the Container Runtime Interface (CRI), and starts the containers.
6. Status Reporting. The kubelet continuously reports Pod status (Pending → Running → Succeeded/Failed) back to the API server, which persists it in etcd.
7. Ongoing Reconciliation. Controllers keep watching. If a container crashes, the kubelet restarts it (based on the Pod's restartPolicy). If a node goes down, the node controller marks it as NotReady, and the ReplicaSet controller creates replacement Pods on healthy nodes.

Seeing It in Practice

You can observe this flow in real time. Open two terminal windows — one to watch events as they happen, and another to trigger the deployment:

# Terminal 1 — watch cluster events in real time
kubectl get events --watch --sort-by='.lastTimestamp'

# Terminal 2 — apply a simple deployment
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-demo
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
EOF

In the events window, you will see the exact sequence play out: the Deployment is created, then a ReplicaSet, then three Pods. Each Pod goes through Scheduled → Pulling → Pulled → Created → Started events. You can also inspect each layer directly:

# See the ownership chain: Deployment → ReplicaSet → Pod
kubectl get deploy nginx-demo -o wide
kubectl get rs -l app=nginx-demo -o wide
kubectl get pods -l app=nginx-demo -o wide

# Check which node each Pod was scheduled to
kubectl get pods -l app=nginx-demo -o custom-columns=\
NAME:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase

Managed Kubernetes vs. Self-Managed Clusters

When you run Kubernetes, you have a choice: manage the control plane yourself, or let a cloud provider handle it. This decision affects your operational burden, cost model, and level of control. In production, most teams choose managed Kubernetes unless they have specific compliance or customization requirements.

Quick Cluster Inspection Commands

Regardless of whether your cluster is managed or self-managed, these commands give you immediate visibility into the architecture of any cluster you connect to:

# Cluster info: API server endpoint, CoreDNS, and add-on URLs
kubectl cluster-info

# All nodes with roles, version, and OS info
kubectl get nodes -o wide

# Control plane components health (self-managed clusters)
kubectl get componentstatuses   # deprecated but still works in many clusters
kubectl get --raw='/healthz?verbose'

# See system Pods running the control plane and node agents
kubectl get pods -n kube-system -o wide

# Detailed view of a specific node's capacity and allocatable resources
kubectl describe node <node-name> | grep -A 6 "Capacity\|Allocatable"

Putting It All Together

The architecture of a Kubernetes cluster is deliberate in its separation of concerns. The API server is the single communication hub. etcd is the single source of truth. The scheduler makes placement decisions. Controllers reconcile desired state with reality. Kubelets do the work of running containers. Every piece is independently replaceable and horizontally scalable.

This architecture is why Kubernetes can self-heal: if a node fails, the node controller detects it, the ReplicaSet controller creates replacement Pods, the scheduler places them on healthy nodes, and the kubelets start them — all automatically, with no human intervention. The next section digs deeper into each control plane component and how they work internally.

Control Plane Components — Under the Hood

The control plane is the brain of a Kubernetes cluster. It makes global decisions about scheduling, detects and responds to cluster events, and serves as the single source of truth for the desired state of every object. Understanding how each component works — and how they interact — is essential for debugging production issues and designing resilient clusters.

This section dissects four components: the API server (the front door), etcd (the memory), the scheduler (the matchmaker), and the controller manager (the reconciliation engine). We will look at internals, trace real request flows, and use kubectl to inspect each component in a live cluster.

flowchart TB
    subgraph ControlPlane["Control Plane"]
        API["kube-apiserver\n(REST API Gateway)"]
        ETCD["etcd\n(Distributed KV Store)"]
        SCHED["kube-scheduler\n(Pod Placement)"]
        CM["kube-controller-manager\n(Reconciliation Loops)"]
    end

    USER["kubectl / Client"] -->|"REST + AuthN/AuthZ\n+ Admission"| API
    API <-->|"gRPC\n(reads & writes)"| ETCD
    API -->|"Watch: unscheduled Pods"| SCHED
    SCHED -->|"Bind Pod → Node"| API
    API -->|"Watch: resource changes"| CM
    CM -->|"Create/Update objects"| API

    subgraph Nodes["Worker Nodes"]
        K1["kubelet"]
        K2["kubelet"]
    end

    API -->|"Watch: Pod specs"| K1
    API -->|"Watch: Pod specs"| K2

    style ControlPlane fill:#1a1a2e,stroke:#4a9eff,stroke-width:2px,color:#e0e0e0
    style Nodes fill:#1a1a2e,stroke:#50c878,stroke-width:2px,color:#e0e0e0
    style API fill:#2d3a5c,stroke:#4a9eff,color:#e0e0e0
    style ETCD fill:#2d3a5c,stroke:#f5a623,color:#e0e0e0
    style SCHED fill:#2d3a5c,stroke:#50c878,color:#e0e0e0
    style CM fill:#2d3a5c,stroke:#c77dba,color:#e0e0e0
    

kube-apiserver — The Front Door to Everything

Every interaction with your cluster — whether from kubectl, the dashboard, a CI/CD pipeline, or an internal controller — goes through the API server. It is the only component that talks to etcd directly. Nothing else reads or writes persistent state without going through this gateway first.

The API server is a stateless REST API built on HTTP/2. You can scale it horizontally by running multiple instances behind a load balancer. Each instance is functionally identical because etcd holds all the state.

The Request Pipeline: AuthN → AuthZ → Admission → etcd

Every request that hits the API server passes through a well-defined pipeline. Understanding this pipeline is critical for debugging 403 Forbidden errors, webhook failures, and mysterious object mutations.

flowchart LR
    REQ["Incoming\nRequest"] --> AUTHN["Authentication\n(Who are you?)"]
    AUTHN --> AUTHZ["Authorization\n(Can you do this?)"]
    AUTHZ --> MUT["Mutating\nAdmission"]
    MUT --> SCHEMA["Object Schema\nValidation"]
    SCHEMA --> VAL["Validating\nAdmission"]
    VAL --> ETCD["Persist\nto etcd"]

    style REQ fill:#2d3a5c,stroke:#4a9eff,color:#e0e0e0
    style AUTHN fill:#2d3a5c,stroke:#50c878,color:#e0e0e0
    style AUTHZ fill:#2d3a5c,stroke:#50c878,color:#e0e0e0
    style MUT fill:#2d3a5c,stroke:#f5a623,color:#e0e0e0
    style SCHEMA fill:#2d3a5c,stroke:#c77dba,color:#e0e0e0
    style VAL fill:#2d3a5c,stroke:#f5a623,color:#e0e0e0
    style ETCD fill:#2d3a5c,stroke:#f5a623,color:#e0e0e0
    

Authentication determines who is making the request. The API server evaluates multiple authenticators in sequence — client certificates, bearer tokens, OIDC tokens, service account tokens — and stops at the first one that succeeds. The result is a username, UID, and group membership.

Authorization determines what the authenticated identity is allowed to do. Kubernetes supports multiple authorizers: Node, ABAC, RBAC, and Webhook. In practice, nearly every cluster uses RBAC. The API server checks each authorizer in order; a single "allow" grants the request, while a "deny" from all authorizers rejects it.

Admission control happens in two phases. Mutating admission webhooks and built-in plugins run first — they can modify the object (e.g., injecting sidecar containers, setting default resource requests). Then validating admission webhooks run — they can accept or reject but cannot modify. This two-phase design means validators always see the final, mutated object.

# Check which admission plugins are enabled on your API server
kubectl get pod kube-apiserver-controlplane -n kube-system \
  -o jsonpath='{.spec.containers[0].command}' | tr ',' '\n' | grep enable-admission

# List all registered mutating and validating webhook configurations
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations

API Groups and Versioning

Kubernetes organizes its API into groups and versions. The core group (Pods, Services, ConfigMaps) lives at /api/v1. Named groups live under /apis/<group>/<version> — for example, /apis/apps/v1 for Deployments, or /apis/batch/v1 for Jobs. Every resource has a stability level indicated by its version: v1 (GA), v1beta1 (beta), or v1alpha1 (alpha).

# Discover all API groups and their preferred versions
kubectl api-versions

# List all resources in the 'apps' group
kubectl api-resources --api-group=apps

# Make a raw API call — bypasses kubectl abstractions
kubectl get --raw /apis/apps/v1/namespaces/default/deployments | jq '.items[].metadata.name'

# Check API server health endpoints
kubectl get --raw /healthz
kubectl get --raw /readyz

etcd — The Cluster's Source of Truth

etcd is a distributed, strongly consistent key-value store. Every Kubernetes object — every Pod, Service, Secret, ConfigMap, and CustomResource — is serialized (typically as protobuf) and stored in etcd. If etcd is lost and unrecoverable, the cluster's desired state is gone. This makes etcd the single most critical piece of infrastructure in any Kubernetes deployment.

Data Model and Key Structure

Kubernetes stores objects in etcd under a predictable key hierarchy. The default prefix is /registry. A Deployment named nginx in the default namespace is stored at /registry/deployments/default/nginx. Understanding this structure is useful when inspecting etcd directly for disaster recovery or debugging.

# List top-level keys in etcd (run on a control plane node)
ETCDCTL_API=3 etcdctl get /registry --prefix --keys-only --limit=20 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Look up a specific object
ETCDCTL_API=3 etcdctl get /registry/deployments/default/nginx \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Raft Consensus

etcd uses the Raft consensus algorithm to replicate data across its cluster members. One member is elected the leader; all writes go through the leader and are replicated to followers. A write is committed only when a majority (quorum) of members acknowledge it. This means a 3-node etcd cluster tolerates 1 failure, and a 5-node cluster tolerates 2 failures.

flowchart LR
    C["API Server\n(Client)"] -->|"Write request"| L["etcd Leader"]
    L -->|"Replicate log entry"| F1["etcd Follower 1"]
    L -->|"Replicate log entry"| F2["etcd Follower 2"]
    F1 -->|"Acknowledge"| L
    F2 -->|"Acknowledge"| L
    L -->|"Commit (quorum reached)\nReturn success"| C

    style C fill:#2d3a5c,stroke:#4a9eff,color:#e0e0e0
    style L fill:#2d3a5c,stroke:#f5a623,color:#e0e0e0
    style F1 fill:#2d3a5c,stroke:#50c878,color:#e0e0e0
    style F2 fill:#2d3a5c,stroke:#50c878,color:#e0e0e0
    

Compaction, Defragmentation, and Backup

etcd keeps a history of all key revisions to support the watch mechanism. Over time, this history grows and consumes disk space. Compaction discards revisions older than a given point, while defragmentation reclaims the freed disk space. Kubernetes runs automatic compaction (default: every 5 minutes, retaining roughly the last 5 minutes of history), but defragmentation must be triggered manually or via a cron job.

# Check etcd cluster health and member list
ETCDCTL_API=3 etcdctl endpoint health --cluster \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

ETCDCTL_API=3 etcdctl member list --write-out=table \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Check current database size and revision
ETCDCTL_API=3 etcdctl endpoint status --write-out=table \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Snapshot backup — THE most important operational task for etcd
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the snapshot is valid
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-20240115.db \
  --write-out=table

# Restore from snapshot (disaster recovery)
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-20240115.db \
  --data-dir=/var/lib/etcd-restored \
  --name=controlplane \
  --initial-cluster=controlplane=https://192.168.1.10:2380 \
  --initial-advertise-peer-urls=https://192.168.1.10:2380

kube-scheduler — Deciding Where Pods Run

When you create a Pod (directly or via a Deployment), it initially has no spec.nodeName. The scheduler watches for these unscheduled Pods, evaluates every available node against a series of constraints and preferences, and then binds the Pod to the best node by writing the node name back to the API server. The kubelet on that node then picks up the Pod and starts its containers.

The Two-Phase Scheduling Cycle

The scheduler operates in two distinct phases for each unscheduled Pod. First, it filters out nodes that cannot run the Pod. Then, it scores the remaining candidates to find the best fit. This filter-then-score approach keeps the algorithm efficient — scoring only runs on nodes that have already passed all hard constraints.

flowchart LR
    P["Unscheduled Pod"] --> F["Filter Phase\n(Predicates)"]
    F -->|"Feasible nodes"| S["Score Phase\n(Priorities)"]
    S -->|"Highest score"| B["Bind to Node"]

    F -.- F1["NodeResourcesFit"]
    F -.- F2["NodeAffinity"]
    F -.- F3["TaintToleration"]
    F -.- F4["PodTopologySpread"]

    S -.- S1["LeastAllocated"]
    S -.- S2["BalancedAllocation"]
    S -.- S3["ImageLocality"]
    S -.- S4["NodeAffinityPriority"]

    style P fill:#2d3a5c,stroke:#4a9eff,color:#e0e0e0
    style F fill:#2d3a5c,stroke:#f5a623,color:#e0e0e0
    style S fill:#2d3a5c,stroke:#50c878,color:#e0e0e0
    style B fill:#2d3a5c,stroke:#c77dba,color:#e0e0e0
    

Resource-Based Scheduling

The scheduler uses requests (not limits) to determine if a node has enough capacity. If a Pod requests 500m CPU and 256Mi memory, the scheduler only considers nodes with at least that much allocatable capacity remaining. This is why setting accurate resource requests is critical — overestimate and you waste capacity, underestimate and Pods get scheduled onto overloaded nodes.

# See allocatable resources vs current requests on each node
kubectl describe nodes | grep -A 5 "Allocated resources"

# More precise: get node capacity, allocatable, and current allocation
kubectl get nodes -o custom-columns=\
  NAME:.metadata.name,\
  CPU_CAP:.status.capacity.cpu,\
  MEM_CAP:.status.capacity.memory,\
  CPU_ALLOC:.status.allocatable.cpu,\
  MEM_ALLOC:.status.allocatable.memory

# Why was a Pod not scheduled? Check events.
kubectl describe pod <pod-name> | grep -A 10 "Events"

# Inspect scheduler decisions in the scheduler logs
kubectl logs -n kube-system kube-scheduler-controlplane --tail=50

Node Affinity and Anti-Affinity

Node affinity lets you constrain which nodes a Pod can be scheduled on based on node labels. It comes in two flavors: requiredDuringSchedulingIgnoredDuringExecution (hard requirement — filter phase) and preferredDuringSchedulingIgnoredDuringExecution (soft preference — scoring phase). The "IgnoredDuringExecution" suffix means the rule only applies at scheduling time; if a node's labels change later, already-running Pods are not evicted.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: gpu-type
                operator: In
                values: ["nvidia-a100", "nvidia-v100"]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 80
          preference:
            matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: ["us-east-1a"]
  containers:
    - name: training
      image: ml-training:v2
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
          nvidia.com/gpu: "1"

Taints and Tolerations

While affinity pulls Pods toward nodes, taints push Pods away. A taint on a node repels all Pods that do not have a matching toleration. This is how Kubernetes keeps user workloads off control plane nodes (they carry a node-role.kubernetes.io/control-plane:NoSchedule taint) and how you can dedicate nodes for specific purposes.

There are three taint effects: NoSchedule (hard — never schedule here without a toleration), PreferNoSchedule (soft — try to avoid), and NoExecute (hard — evict already-running Pods that do not tolerate the taint).

# View taints on all nodes
kubectl get nodes -o custom-columns=\
  NAME:.metadata.name,\
  TAINTS:.spec.taints

# Add a taint to dedicate nodes for a team
kubectl taint nodes worker-3 team=ml:NoSchedule

# Remove a taint (note the trailing dash)
kubectl taint nodes worker-3 team=ml:NoSchedule-

# Check which Pods tolerate control-plane taints
kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.spec.tolerations[]? | .key == "node-role.kubernetes.io/control-plane") |
  "\(.metadata.namespace)/\(.metadata.name)"'

# Pod that tolerates the ML team taint
apiVersion: v1
kind: Pod
metadata:
  name: ml-job
spec:
  tolerations:
    - key: "team"
      operator: "Equal"
      value: "ml"
      effect: "NoSchedule"
  containers:
    - name: training
      image: pytorch:latest

kube-controller-manager — Relentless Reconciliation

The controller manager is a single binary that bundles dozens of independent control loops (controllers). Each controller watches a specific set of resources through the API server and continuously drives the actual state toward the desired state. This is the heart of the Kubernetes declarative model: you declare what you want, and controllers make it happen.

The Controller Pattern

Every controller follows the same fundamental loop: observe (watch the API server for changes), diff (compare desired state vs. actual state), and act (make API calls to close the gap). If a controller crashes and restarts, it simply re-reads the current state from the API server and picks up where it left off. No state is stored locally.

flowchart TB
    WATCH["Watch API Server\n(Informer Cache)"] --> DIFF["Compare Desired\nvs. Actual State"]
    DIFF -->|"Drift detected"| ACT["Take Action\n(Create / Update / Delete)"]
    ACT --> API["API Server"]
    API --> WATCH

    DIFF -->|"In sync"| WATCH

    style WATCH fill:#2d3a5c,stroke:#4a9eff,color:#e0e0e0
    style DIFF fill:#2d3a5c,stroke:#f5a623,color:#e0e0e0
    style ACT fill:#2d3a5c,stroke:#50c878,color:#e0e0e0
    style API fill:#2d3a5c,stroke:#c77dba,color:#e0e0e0
    

Key Controllers and What They Do

# Observe the controller pattern in action — scale a Deployment
# and watch the ReplicaSet controller respond
kubectl scale deployment/nginx --replicas=5
kubectl get events --watch --field-selector reason=SuccessfulCreate

# See the ownership chain: Deployment → ReplicaSet → Pod
kubectl get rs -o custom-columns=\
  NAME:.metadata.name,\
  OWNER:.metadata.ownerReferences[0].name,\
  DESIRED:.spec.replicas,\
  READY:.status.readyReplicas

# Inspect controller manager logs
kubectl logs -n kube-system kube-controller-manager-controlplane --tail=100

# Check which controllers are enabled
kubectl get pod kube-controller-manager-controlplane -n kube-system \
  -o jsonpath='{.spec.containers[0].command}' | tr ',' '\n' | grep controllers

Leader Election

In a highly available cluster with multiple control plane nodes, you have multiple instances of the controller manager and scheduler running. But only one instance of each should be actively reconciling at any time — otherwise you'd get duplicate actions. Kubernetes solves this with leader election.

Each component races to acquire a Lease object in the kube-system namespace. The winner becomes the active leader and periodically renews the lease. If the leader crashes, it stops renewing, and another instance acquires the lease within seconds. You can inspect the current leaders directly.

# Check which node is the current leader for each component
kubectl get lease -n kube-system

# Detailed view — see holderIdentity and renewal time
kubectl get lease kube-controller-manager -n kube-system -o yaml
kubectl get lease kube-scheduler -n kube-system -o yaml

# Example output (holderIdentity shows the current leader):
# holderIdentity: controlplane-1_xxxxxxxx-xxxx
# leaseDurationSeconds: 15
# renewTime: "2024-01-15T10:23:45.000000Z"

Putting It All Together: The Life of a Deployment

To solidify how these components interact, let's trace what happens end-to-end when you run kubectl apply -f deployment.yaml for a new 3-replica Deployment.

# Watch the entire chain in real-time in a second terminal
kubectl get events --watch

# Then trigger a Deployment in your first terminal
kubectl create deployment nginx --image=nginx:1.25 --replicas=3

# You will see events flow through:
#   Deployment created
#   ReplicaSet created (by deployment-controller)
#   Pods created (by replicaset-controller)
#   Pods scheduled (by default-scheduler)
#   Containers pulling image, started (by kubelet)

Node Components — Kubelet, Kube-Proxy, and Container Runtime

The control plane makes decisions, but it is the node components that do the actual work. Every worker node in a Kubernetes cluster runs three core components: the kubelet, which manages pod lifecycle; kube-proxy, which handles network routing for Services; and a container runtime, which pulls images and runs containers. Understanding how these three collaborate is essential for debugging node-level issues and optimizing cluster performance.

Each of these components operates independently but communicates through well-defined interfaces. The kubelet talks to the container runtime via the Container Runtime Interface (CRI). Kube-proxy watches the API server for Service and Endpoint changes, then programs kernel-level routing rules. The container runtime handles the low-level mechanics of creating and destroying containers. Together, they turn a bare Linux machine into a functioning Kubernetes node.

Worker Node Architecture

The following diagram shows how the three node components interact on a single worker node when traffic arrives for a Kubernetes Service and when the kubelet manages a pod's lifecycle.

flowchart TB
    API["API Server\n(Control Plane)"]

    subgraph WorkerNode["Worker Node"]
        direction TB
        KL["kubelet"]
        KP["kube-proxy"]
        CR["Container Runtime\n(containerd / CRI-O)"]
        cAdvisor["cAdvisor\n(embedded)"]
        IPTABLES["iptables / IPVS\n(kernel)"]

        subgraph Pod1["Pod A"]
            C1["Container 1"]
            C2["Container 2"]
        end

        subgraph Pod2["Pod B"]
            C3["Container 3"]
        end
    end

    API -- "pod specs,\ndesired state" --> KL
    KL -- "status reports,\nnode heartbeat" --> API
    KL -- "CRI gRPC calls\n(RunPodSandbox,\nCreateContainer)" --> CR
    CR --> Pod1
    CR --> Pod2
    KL -- "metrics" --> cAdvisor
    cAdvisor -.-> Pod1
    cAdvisor -.-> Pod2
    API -- "Service &\nEndpointSlice watches" --> KP
    KP -- "programs rules" --> IPTABLES
    IPTABLES -- "DNAT to\npod IP:port" --> Pod1
    IPTABLES -- "DNAT to\npod IP:port" --> Pod2
    

Kubelet — The Node Agent

The kubelet is a daemon that runs on every node (including control plane nodes). It is the sole authority responsible for ensuring that the containers described in a Pod spec are running and healthy. The kubelet does not manage containers that were not created by Kubernetes — it only cares about Pods assigned to its node by the API server (or defined as static pods on disk).

Pod Lifecycle Management

When the scheduler assigns a Pod to a node, the kubelet picks up the assignment by watching the API server. It then drives the Pod through its lifecycle: pulling images, creating the pod sandbox (network namespace), starting init containers in sequence, starting app containers, running startup/liveness/readiness probes, and ultimately tearing everything down when the Pod is deleted. Each transition is reported back to the API server as a pod status update.

The kubelet's main sync loop runs every 10 seconds by default (configurable via --sync-frequency). On each iteration, it compares the desired state from the API server against the actual state on the node and reconciles any differences. If a container crashes, the kubelet restarts it according to the pod's restartPolicy with an exponential backoff that caps at 5 minutes.

You can inspect the kubelet's view of what is running on a node by querying its read-only status port or using kubectl:

# List all pods running on a specific node
kubectl get pods --all-namespaces --field-selector spec.nodeName=worker-1

# Check kubelet health on the node itself
systemctl status kubelet

# View kubelet logs for pod lifecycle events
journalctl -u kubelet --no-pager --since "10 minutes ago" | grep -i "SyncLoop"

# Inspect the kubelet's configuration
kubectl get --raw "/api/v1/nodes/worker-1/proxy/configz" | jq .

Static Pods

Static pods are managed directly by the kubelet on a specific node, without the API server scheduling them. The kubelet watches a directory on disk (default: /etc/kubernetes/manifests/) and automatically creates or destroys pods when YAML files are added to or removed from that directory. This is how kubeadm-based clusters run control plane components — etcd, kube-apiserver, kube-controller-manager, and kube-scheduler all run as static pods.

The kubelet creates a mirror pod in the API server for each static pod so that kubectl get pods can show them. However, you cannot delete a mirror pod through the API — you must remove the manifest file from disk.

# Find the static pod manifest directory
ps aux | grep kubelet | grep -- "--pod-manifest-path"

# Or check the kubelet config file
cat /var/lib/kubelet/config.yaml | grep staticPodPath

# List static pod manifests (on a kubeadm control plane node)
ls -la /etc/kubernetes/manifests/
# etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

You can create your own static pods by placing a manifest in the static pod directory. The kubelet will pick it up within seconds:

# /etc/kubernetes/manifests/debug-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: debug-static
  namespace: default
spec:
  containers:
  - name: debug
    image: busybox:1.36
    command: ["sleep", "3600"]

CRI — Container Runtime Interface

The kubelet does not start containers directly. Instead, it communicates with the container runtime through the Container Runtime Interface (CRI), a gRPC-based API defined as a set of protobuf services. CRI has two services: the RuntimeService (for managing pod sandboxes and containers) and the ImageService (for pulling, listing, and removing images).

This abstraction is what allows Kubernetes to support multiple container runtimes interchangeably. The kubelet connects to the CRI endpoint via a Unix socket — typically /run/containerd/containerd.sock for containerd or /var/run/crio/crio.sock for CRI-O.

# Check which CRI endpoint the kubelet is using
ps aux | grep kubelet | grep -- "container-runtime-endpoint"

# Use crictl to interact with the runtime directly (works with any CRI runtime)
crictl --runtime-endpoint unix:///run/containerd/containerd.sock info

# List running containers through CRI
crictl ps

# List pod sandboxes
crictl pods

# Pull an image through CRI
crictl pull nginx:1.25

# Inspect a specific container
crictl inspect <CONTAINER_ID>

cAdvisor Metrics and Resource Monitoring

The kubelet embeds cAdvisor (Container Advisor), which collects CPU, memory, filesystem, and network usage metrics for every running container. These metrics serve two critical purposes: they power the kubectl top command (via the Metrics Server, which scrapes the kubelet's /metrics/resource endpoint), and they inform the kubelet's own eviction decisions when a node runs low on resources.

# View resource usage per pod on a node (requires Metrics Server)
kubectl top pods -n default

# View resource usage per node
kubectl top nodes

# Query the kubelet's metrics endpoint directly (from the node)
curl -sk https://localhost:10250/metrics/resource \
  --cert /var/lib/kubelet/pki/kubelet-client-current.pem \
  --key /var/lib/kubelet/pki/kubelet-client-current.pem | head -30

# Check kubelet's summary API for detailed per-container stats
kubectl get --raw "/api/v1/nodes/worker-1/proxy/stats/summary" | jq '.pods[0]'

Garbage Collection

Over time, terminated containers and unused images accumulate on each node, consuming disk space. The kubelet runs garbage collection for both. Container garbage collection removes dead containers based on three settings: MaxPerPodContainer (default 1 — keep the last terminated container per pod), MaxContainers (total dead containers on the node), and a minimum age threshold. Image garbage collection triggers based on disk usage thresholds.

# Key kubelet config fields for garbage collection (/var/lib/kubelet/config.yaml)
imageMinimumGCAge: 2m           # Don't GC images younger than 2 minutes
imageGCHighThresholdPercent: 85 # Start removing images when disk hits 85%
imageGCLowThresholdPercent: 80  # Stop removing images when disk drops to 80%
evictionHard:
  imagefs.available: "15%"      # Evict pods if image filesystem < 15% free
  memory.available: "100Mi"     # Evict pods if available memory < 100Mi
  nodefs.available: "10%"       # Evict pods if root filesystem < 10% free

# Check current image disk usage on a node
crictl images | tail -5
crictl imagefsinfo

# Manually trigger image cleanup (remove unused images)
crictl rmi --prune

# See if the node is under disk pressure (triggers eviction)
kubectl describe node worker-1 | grep -A 5 "Conditions"

Kube-Proxy — Service Networking at the Kernel Level

When you create a Kubernetes Service, it gets a stable virtual IP address (ClusterIP) that doesn't map to any network interface. Kube-proxy is the component that makes this virtual IP actually work. It runs on every node, watches the API server for Service and EndpointSlice objects, and programs the node's kernel networking stack to translate Service IPs into real pod IPs using destination NAT (DNAT).

Kube-proxy does not proxy traffic itself in the data path (despite its name). It is a control plane component for node networking — it configures the rules and then steps aside. All actual packet forwarding happens in the Linux kernel.

How Service Routing Works

flowchart LR
    Client["Client Pod\n10.244.1.5"]
    SVC["Service ClusterIP\n10.96.0.100:80"]
    KERNEL["Linux Kernel\n(iptables / IPVS)"]
    EP1["Pod A\n10.244.2.10:8080"]
    EP2["Pod B\n10.244.3.22:8080"]
    EP3["Pod C\n10.244.1.18:8080"]

    WATCH["kube-proxy\n(watches API server)"]
    APISVR["API Server"]

    Client -- "dst: 10.96.0.100:80" --> KERNEL
    KERNEL -- "DNAT to\n10.244.2.10:8080" --> EP1
    KERNEL -. "or" .-> EP2
    KERNEL -. "or" .-> EP3
    APISVR -- "Service &\nEndpointSlice changes" --> WATCH
    WATCH -- "programs\nrouting rules" --> KERNEL
    

The flow works like this: a client pod sends a packet to the Service ClusterIP. The kernel intercepts the packet before it leaves the node (using PREROUTING or OUTPUT chains) and rewrites the destination address to one of the backing pod IPs. The response packet goes directly back to the client with the source address rewritten to the Service IP, so the client never knows which pod it hit.

Proxy Modes Compared

Kube-proxy supports three modes for programming these routing rules. The mode determines which kernel subsystem does the actual work. Each has meaningful trade-offs at scale.

Inspecting Kube-Proxy Rules

Understanding what kube-proxy has actually programmed into the kernel is one of the most valuable debugging skills for Service connectivity issues. The commands differ by proxy mode.

# Check which proxy mode is active
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

# --- iptables mode ---
# List all KUBE-SERVICES chains (one per ClusterIP:port)
iptables -t nat -L KUBE-SERVICES -n | head -20

# Trace rules for a specific Service ClusterIP
iptables -t nat -L -n -v | grep "10.96.0.100"

# Show the full DNAT chain for a service
# (Follow KUBE-SVC-xxx -> KUBE-SEP-xxx chains to see pod endpoints)
iptables -t nat -L KUBE-SVC-XXXXXX -n -v

# --- IPVS mode ---
# List all virtual servers (Service IPs) and their real servers (Pod IPs)
ipvsadm -Ln

# Show stats for a specific Service ClusterIP
ipvsadm -Ln -t 10.96.0.100:80

# --- nftables mode ---
# List nftables rules managed by kube-proxy
nft list table ip kube-proxy

Session Affinity

By default, kube-proxy distributes traffic across all healthy endpoints with no stickiness. If you need a client to consistently reach the same backend pod (for example, for in-memory sessions), you can enable session affinity on the Service. Kubernetes supports ClientIP affinity, which routes all requests from the same source IP to the same pod for a configurable timeout (default: 10,800 seconds / 3 hours).

apiVersion: v1
kind: Service
metadata:
  name: web-app
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 1800  # 30 minutes

# Verify session affinity is set on a Service
kubectl describe svc web-app | grep -i "session"

# Test affinity — repeated requests should hit the same pod
for i in $(seq 1 5); do
  kubectl exec test-pod -- curl -s web-app.default.svc.cluster.local | grep "pod-name"
done

Kube-Proxy Logs and Debugging

# kube-proxy usually runs as a DaemonSet — find the pod on your node
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

# View kube-proxy logs
kubectl logs -n kube-system kube-proxy-abc12 --tail=50

# Check kube-proxy's current configuration
kubectl get configmap kube-proxy -n kube-system -o yaml

# Verify kube-proxy is correctly watching EndpointSlices
kubectl logs -n kube-system kube-proxy-abc12 | grep -i "endpoints"

# Check conntrack entries for active NAT translations
conntrack -L -d 10.96.0.100 2>/dev/null | head -10

Container Runtime — Where Containers Actually Run

The container runtime is the software that actually executes containers. It pulls images from registries, creates filesystem layers, sets up namespaces and cgroups, and starts the process inside the container. Kubernetes interacts with the runtime exclusively through the CRI specification, which means any runtime implementing the CRI gRPC API works with Kubernetes.

The CRI Specification

CRI defines two gRPC services. The RuntimeService handles the pod and container lifecycle — creating sandboxes, starting and stopping containers, executing commands, attaching to running containers, and port forwarding. The ImageService handles image operations — pulling, listing, inspecting, and removing images. This clean separation means the kubelet never needs to know the implementation details of how containers or images are managed under the hood.

containerd — The Default Runtime

containerd is the default container runtime for most Kubernetes distributions, including kubeadm clusters, GKE, EKS, and AKS. It was originally extracted from Docker as a standalone daemon and donated to the CNCF. containerd handles the full container lifecycle: image transfer and storage, container execution (via runc as the low-level OCI runtime), snapshotting (filesystem layering), and network namespace management.

The architecture is layered: the kubelet calls containerd's CRI plugin over gRPC, containerd manages image and container metadata, and it delegates actual container creation to an OCI runtime (usually runc, but gVisor's runsc and Kata Containers' kata-runtime are also supported).

# Check containerd status
systemctl status containerd

# View containerd version and runtime info
ctr version

# Use the containerd-specific CLI to list containers (in the k8s.io namespace)
ctr -n k8s.io containers list

# List images managed by containerd for Kubernetes
ctr -n k8s.io images list | head -10

# View the containerd configuration
cat /etc/containerd/config.toml | grep -A 5 "plugins.*cri"

# Check which OCI runtime containerd is using
cat /etc/containerd/config.toml | grep -A 3 "runtimes.runc"

CRI-O — The Kubernetes-Native Alternative

CRI-O is a lightweight container runtime built specifically for Kubernetes. Unlike containerd, which has a broader scope (Docker, Swarm, etc.), CRI-O implements only the CRI interface and nothing more. It follows a strict version-locking policy with Kubernetes: CRI-O 1.29.x supports Kubernetes 1.29.x, CRI-O 1.30.x supports Kubernetes 1.30.x, and so on. This makes it a popular choice in OpenShift clusters and environments that want the thinnest possible runtime layer.

# Check CRI-O status
systemctl status crio

# View CRI-O version and config
crio --version
crio config --default | head -30

# On a CRI-O node, crictl works exactly the same way
crictl --runtime-endpoint unix:///var/run/crio/crio.sock ps
crictl --runtime-endpoint unix:///var/run/crio/crio.sock images

The Dockershim Deprecation

Before Kubernetes 1.24, the kubelet could talk to Docker Engine via a built-in adapter called dockershim. This adapter translated CRI calls into Docker API calls. The chain was wasteful: kubelet → dockershim → Docker Engine → containerd → runc. Kubernetes 1.24 removed dockershim entirely. If your cluster previously used Docker as its runtime, it was migrated to containerd (which Docker itself used internally). Your container images remain 100% compatible because both Docker and Kubernetes build and run OCI-compliant images.

Image Pulling and Caching

When a pod is scheduled to a node, the kubelet instructs the container runtime to pull any images not already present on the node. Images are stored in a content-addressable store and shared across containers — if two pods use the same image, it is stored only once. The pull behavior is controlled by the pod spec's imagePullPolicy.

# List images cached on the node
crictl images

# Check the disk space used by images
crictl imagefsinfo

# Pre-pull an image to avoid cold-start latency
crictl pull gcr.io/my-project/my-app:v2.1.0

# Debug image pull failures — check kubelet logs
journalctl -u kubelet --no-pager --since "5 minutes ago" | grep -i "pull"

# Check events on a pod stuck in ImagePullBackOff
kubectl describe pod my-pod | grep -A 10 "Events"

Putting It All Together — Node Status and Debugging

The kubelet reports the overall health of all three components back to the API server as node conditions. When any component is unhealthy, it surfaces as a condition change on the node object. Understanding these conditions is the starting point for diagnosing node-level problems.

# Full node inspection — allocatable resources, conditions, images, and running pods
kubectl describe node worker-1

# Check node conditions specifically
kubectl get node worker-1 -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'

# Verify the container runtime the node is using
kubectl get node worker-1 -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'
# Output: containerd://1.7.11

# Comprehensive node debug — creates a privileged debug pod on the node
kubectl debug node/worker-1 -it --image=ubuntu:22.04
# Once inside the debug pod:
#   chroot /host
#   systemctl status kubelet
#   systemctl status containerd
#   journalctl -u kubelet --since "30 minutes ago" | tail -50

With a solid understanding of these three components, you can trace any node-level issue from symptom to root cause. A pod stuck in ContainerCreating? Check the kubelet logs and the container runtime. A Service not reachable? Inspect kube-proxy's iptables or IPVS rules. An ImagePullBackOff? Look at the kubelet events and test the image pull with crictl. These components are the foundation — everything else in Kubernetes runs on top of them.

The Kubernetes API and kubectl Essentials

Every interaction with a Kubernetes cluster — whether you type a kubectl command, deploy from CI/CD, or a controller reconciles state — goes through the Kubernetes API server. The API is the single source of truth. Understanding its structure is not optional background knowledge; it is the foundation for everything else you will do with Kubernetes.

kubectl is the primary CLI client for this API. It translates your commands into HTTP requests against the API server, then formats the responses for your terminal. This section covers the API's structure, how to explore it, and the essential kubectl commands organized by real-world workflow.

The Kubernetes API Structure

The Kubernetes API is a RESTful HTTP API organized into API groups. Each group contains a set of related resources, and each resource has one or more supported versions. When you create a Deployment, you are making an HTTP POST to a specific URL path that encodes the group, version, and resource type.

API Groups

Resources are organized into groups to keep the API modular and allow independent versioning. The core group (also called the legacy group) has no explicit group name in its URL path — its resources live directly under /api/v1. All other groups are under /apis/<group-name>/<version>.

The full API URL for a namespaced resource follows this pattern: /apis/{group}/{version}/namespaces/{namespace}/{resource}. For example, to list Deployments in the production namespace, the API server handles a GET request to /apis/apps/v1/namespaces/production/deployments.

Resource Versioning

Each API group version indicates its stability level. This is not just a label — it carries guarantees about backward compatibility and how long the version will be supported.

You specify the API version in every manifest's apiVersion field. For core group resources, you write apiVersion: v1. For named groups, you write apiVersion: group/version — for example, apiVersion: apps/v1 for a Deployment.

Exploring the API with kubectl

You do not need to memorize the full API table. Two commands let you explore it interactively. kubectl api-resources lists every resource type available in your cluster, and kubectl explain gives you inline documentation for any resource or field.

# List all available resource types with their API group and kind
kubectl api-resources

# Filter to only namespaced resources
kubectl api-resources --namespaced=true

# Filter by API group
kubectl api-resources --api-group=apps

# Show supported API versions
kubectl api-versions

The output of api-resources shows each resource's short name (useful for saving keystrokes), the API group it belongs to, whether it is namespaced, and its Kind. For example, you will see that deploy is the short name for deployments in the apps group.

# Get top-level documentation for a resource
kubectl explain deployment

# Drill into a specific field path
kubectl explain deployment.spec.strategy

# Show the full recursive structure
kubectl explain deployment.spec --recursive

# Specify an API version explicitly
kubectl explain cronjob --api-version=batch/v1

Kubeconfig, Contexts, and Namespace Management

Before you can talk to a cluster, kubectl needs to know which cluster, which user credentials, and which namespace to use by default. All of this is stored in the kubeconfig file, typically located at ~/.kube/config. The file has three main sections: clusters, users, and contexts.

Kubeconfig Structure

apiVersion: v1
kind: Config
current-context: dev-cluster

clusters:
- name: dev-cluster
  cluster:
    server: https://dev-k8s.example.com:6443
    certificate-authority-data: LS0tLS1CRUd...
- name: prod-cluster
  cluster:
    server: https://prod-k8s.example.com:6443
    certificate-authority-data: LS0tLS1CRUd...

users:
- name: dev-admin
  user:
    client-certificate-data: LS0tLS1CRUd...
    client-key-data: LS0tLS1CRUd...
- name: prod-reader
  user:
    token: eyJhbGciOiJSUzI1NiIs...

contexts:
- name: dev-cluster
  context:
    cluster: dev-cluster
    user: dev-admin
    namespace: default
- name: prod-readonly
  context:
    cluster: prod-cluster
    user: prod-reader
    namespace: monitoring

A context is a triple of (cluster, user, namespace). It gives you a named shortcut so you can switch between environments with a single command instead of specifying credentials and endpoints every time.

Context Switching

# See all available contexts
kubectl config get-contexts

# Check which context is currently active
kubectl config current-context

# Switch to a different context
kubectl config use-context prod-readonly

# Set the default namespace for the current context
kubectl config set-context --current --namespace=kube-system

# View the full merged kubeconfig
kubectl config view

Namespace Management

Namespaces provide scope for resource names and are the primary boundary for access control policies and resource quotas. Every namespaced command defaults to the namespace set in your current context (or default if none is set). You can override this on any command with the -n flag.

# List all namespaces
kubectl get namespaces

# Run a command in a specific namespace
kubectl get pods -n kube-system

# Query across ALL namespaces
kubectl get pods --all-namespaces
kubectl get pods -A   # shorthand

# Create a new namespace
kubectl create namespace staging

Essential kubectl Commands by Workflow

Rather than listing commands alphabetically, let's organize them by the workflow stages you will actually use every day: create, inspect, update, delete, and debug.

Creating Resources

There are two primary ways to create resources: kubectl apply (declarative) and kubectl create (imperative). The declarative approach with apply is strongly preferred for anything beyond quick experiments, because it tracks the desired state and enables repeatable updates. The next section on the declarative model covers the "why" in depth.

# Declarative: apply a manifest file (create or update)
kubectl apply -f deployment.yaml

# Apply an entire directory of manifests
kubectl apply -f ./k8s/

# Apply from a URL
kubectl apply -f https://raw.githubusercontent.com/org/repo/main/manifests/app.yaml

# Dry-run to validate without creating anything
kubectl apply -f deployment.yaml --dry-run=client
kubectl apply -f deployment.yaml --dry-run=server  # server-side validation

# Imperative: create resources directly (errors if resource already exists)
kubectl create deployment nginx --image=nginx:1.27 --replicas=3
kubectl create service clusterip nginx --tcp=80:80
kubectl create configmap app-config --from-file=config.properties
kubectl create secret generic db-creds --from-literal=password=s3cret

Inspecting Resources

Inspection is where you will spend most of your kubectl time. The core commands are get (list resources), describe (detailed view with events), and logs (container output). Each serves a different purpose in your investigation workflow.

# List pods with status information
kubectl get pods
kubectl get pods -o wide            # includes Node, IP, and nominated node
kubectl get pods --show-labels       # show all labels as columns
kubectl get pods -l app=frontend     # filter by label selector
kubectl get pods --field-selector=status.phase=Running

# List multiple resource types at once
kubectl get pods,services,deployments

# Describe gives you the full picture: spec, status, conditions, and events
kubectl describe pod nginx-7d6b8f5c9-x2k4m
kubectl describe deployment frontend
kubectl describe node worker-01

# Container logs
kubectl logs nginx-7d6b8f5c9-x2k4m
kubectl logs nginx-7d6b8f5c9-x2k4m -c sidecar   # specific container
kubectl logs nginx-7d6b8f5c9-x2k4m --previous     # logs from crashed container
kubectl logs -f nginx-7d6b8f5c9-x2k4m              # follow (stream) logs
kubectl logs -l app=frontend --all-containers       # logs from all matching pods

# Execute commands inside a running container
kubectl exec nginx-7d6b8f5c9-x2k4m -- ls /etc/nginx
kubectl exec -it nginx-7d6b8f5c9-x2k4m -- /bin/sh  # interactive shell

The difference between get and describe is critical. get gives you a tabular summary — perfect for scanning across many resources. describe gives you the full story of a single resource, including the Events section at the bottom, which is often the first place you find the reason for a failure (image pull errors, insufficient resources, failed health checks).

Updating Resources

For updates, the declarative approach is to modify your YAML file and run kubectl apply again. But kubectl also provides imperative update commands that are useful for quick changes during development or incident response.

# Edit a resource in your default editor (opens YAML in $EDITOR)
kubectl edit deployment frontend

# Patch a resource with a JSON merge patch
kubectl patch deployment frontend \
  -p '{"spec":{"replicas":5}}'

# Strategic merge patch (default) — merges arrays intelligently
kubectl patch deployment frontend --type=strategic \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"app","image":"app:v2.1"}]}}}}'

# JSON patch — precise array operations
kubectl patch deployment frontend --type=json \
  -p '[{"op":"replace","path":"/spec/replicas","value":5}]'

# Quick replica scaling
kubectl scale deployment frontend --replicas=10

# Update the container image directly
kubectl set image deployment/frontend app=myapp:v2.1

# Add or update labels and annotations
kubectl label pods nginx-7d6b8f5c9-x2k4m env=staging
kubectl annotate deployment frontend team=platform

Deleting Resources

Deletion can be targeted or broad. By default, kubectl delete waits for the resource to be fully terminated (which respects graceful shutdown periods). Use --wait=false to return immediately.

# Delete a specific resource
kubectl delete pod nginx-7d6b8f5c9-x2k4m
kubectl delete deployment frontend

# Delete using the same manifest file used to create it
kubectl delete -f deployment.yaml

# Delete by label selector
kubectl delete pods -l app=frontend

# Delete all resources of a type in a namespace
kubectl delete pods --all -n staging

# Force-delete a stuck pod (skips graceful shutdown)
kubectl delete pod stuck-pod --grace-period=0 --force

# Delete a namespace and ALL resources within it
kubectl delete namespace staging

Debugging

When things go wrong — and they will — kubectl provides tools to dig deeper without modifying production workloads. Port-forwarding and proxying let you reach cluster-internal endpoints from your local machine. The debug command lets you attach diagnostic containers to running pods.

# Forward a local port to a pod (access pod's port 8080 at localhost:8080)
kubectl port-forward pod/frontend-5d7b8c9f-k2m4x 8080:8080

# Forward to a service (kubectl picks a backing pod)
kubectl port-forward svc/frontend 8080:80

# Start a proxy to the entire API server (accessible at localhost:8001)
kubectl proxy

# Attach a debug container to a running pod (ephemeral container)
kubectl debug -it frontend-5d7b8c9f-k2m4x --image=busybox --target=app

# Create a copy of a pod with a debug container for troubleshooting
kubectl debug frontend-5d7b8c9f-k2m4x -it --copy-to=debug-pod --image=ubuntu

# Debug a node directly (creates a privileged pod on the node)
kubectl debug node/worker-01 -it --image=ubuntu

Output Formatting

The default table output is fine for quick glances, but real-world workflows often require structured data — piping to jq, feeding into scripts, or extracting a single value for a CI variable. The -o flag controls the output format.

Standard Formats

# Full YAML output — great for seeing the complete resource spec
kubectl get deployment frontend -o yaml

# Full JSON output — ideal for piping to jq
kubectl get pods -o json | jq '.items[].metadata.name'

# Wide table with extra columns (Node, IP, etc.)
kubectl get pods -o wide

# Just the resource names (useful for scripting)
kubectl get pods -o name

JSONPath — Extracting Specific Fields

JSONPath expressions let you extract exactly the fields you need without an external JSON processor. The syntax uses curly braces around the path expression, starting from the root object.

# Get the IP address of a specific pod
kubectl get pod nginx-7d6b8f5c9-x2k4m \
  -o jsonpath='{.status.podIP}'

# List all container images across all pods
kubectl get pods \
  -o jsonpath='{.items[*].spec.containers[*].image}'

# Format with newlines for readability
kubectl get pods \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'

# Get the node port of a NodePort service
kubectl get svc frontend \
  -o jsonpath='{.spec.ports[0].nodePort}'

# Extract a secret value (base64-encoded)
kubectl get secret db-creds \
  -o jsonpath='{.data.password}' | base64 -d

Custom Columns — Readable Tables from Arbitrary Fields

When you want a table format but with different columns than the default, custom-columns lets you define exactly what to show. Each column has a header and a JSONPath expression.

# Custom columns: pod name, node, status, and restart count
kubectl get pods -o custom-columns=\
'NAME:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase,RESTARTS:.status.containerStatuses[0].restartCount'

# Deployments with image and replica info
kubectl get deployments -o custom-columns=\
'NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image,DESIRED:.spec.replicas,AVAILABLE:.status.availableReplicas'

Real-World Workflow Patterns

Individual commands are building blocks. In practice, you chain them together to accomplish real tasks. Here are patterns you will use regularly.

Investigating a Failing Deployment

When a deployment is not reaching its desired replica count, work through the resource hierarchy: Deployment → ReplicaSet → Pod → Container logs → Events.

# Step 1: Check the deployment status
kubectl get deployment frontend
kubectl describe deployment frontend | tail -20

# Step 2: Check the ReplicaSet it created
kubectl get replicasets -l app=frontend

# Step 3: Find pods that are not Ready
kubectl get pods -l app=frontend --field-selector=status.phase!=Running

# Step 4: Describe the failing pod for events and conditions
kubectl describe pod frontend-5d7b8c9f-crash1

# Step 5: Check container logs (including previous crashed container)
kubectl logs frontend-5d7b8c9f-crash1 --previous

# Step 6: Check cluster-wide events sorted by time
kubectl get events --sort-by='.lastTimestamp' -n default | tail -20

Quick Rollback After a Bad Deploy

# View rollout history
kubectl rollout history deployment/frontend

# Check the current rollout status
kubectl rollout status deployment/frontend

# Roll back to the previous version
kubectl rollout undo deployment/frontend

# Roll back to a specific revision
kubectl rollout undo deployment/frontend --to-revision=3

# Pause a rollout (prevent further changes from progressing)
kubectl rollout pause deployment/frontend

# Resume a paused rollout
kubectl rollout resume deployment/frontend

Generating Manifests Without Memorizing YAML

One of the most practical kubectl tricks: use imperative commands with --dry-run=client -o yaml to generate manifest templates. This saves you from writing YAML from scratch and ensures the basic structure is correct.

# Generate a Deployment manifest
kubectl create deployment web --image=nginx:1.27 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

# Generate a Service manifest
kubectl create service clusterip web --tcp=80:8080 \
  --dry-run=client -o yaml > service.yaml

# Generate a Job manifest
kubectl create job backup --image=postgres:16 \
  -- pg_dump -h db myapp \
  --dry-run=client -o yaml > job.yaml

# Generate a CronJob manifest
kubectl create cronjob cleanup --image=busybox \
  --schedule="0 2 * * *" -- /bin/sh -c "echo cleaning up" \
  --dry-run=client -o yaml > cronjob.yaml

Working Across Multiple Resources

# Get a comprehensive view of everything in a namespace
kubectl get all -n production

# Watch resources in real time (updates as state changes)
kubectl get pods -w

# Get top resource consumers (requires metrics-server)
kubectl top pods --sort-by=memory
kubectl top nodes

# Diff local changes against the live cluster state before applying
kubectl diff -f deployment.yaml

# Apply changes only when the diff looks right
kubectl diff -f deployment.yaml && kubectl apply -f deployment.yaml

# Bulk operations: restart all pods in a deployment (rolling restart)
kubectl rollout restart deployment/frontend

# Copy files to/from a container
kubectl cp ./local-file.txt frontend-pod:/tmp/remote-file.txt
kubectl cp frontend-pod:/var/log/app.log ./app.log

Quick Reference: Common Short Names and Flags

Typing full resource names gets tedious. Kubernetes provides short names for frequently used resources, and kubectl supports shorthand flags. Here are the ones you will use most.

The Declarative Model and Reconciliation Loops

Most infrastructure tools ask you to write a script of commands: "create this, then modify that, then delete the other thing." Kubernetes works differently. You hand the cluster a document that says what you want, and a fleet of controllers figure out how to make it happen. This is the declarative model, and it is the single most important design decision in Kubernetes.

The declarative approach turns cluster management into a convergence problem. You declare a desired state \u2014 three replicas of your web server, a load balancer on port 443, a 10Gi persistent volume \u2014 and the system continuously works to make reality match that declaration. If something drifts (a pod crashes, a node goes down, someone manually deletes a resource), the controllers detect the gap and close it automatically.

Imperative vs. Declarative: Two Mental Models

Kubernetes supports both imperative and declarative workflows through kubectl, but they represent fundamentally different ways of thinking about infrastructure. Understanding the distinction is critical to using Kubernetes effectively.

Imperative commands tell Kubernetes exactly what to do right now. You issue a verb \u2014 create, scale, delete \u2014 and the action executes immediately. There is no record of intent beyond the resulting object in the cluster.

# Imperative: issue commands one at a time
kubectl create deployment nginx --image=nginx:1.27
kubectl scale deployment nginx --replicas=3
kubectl set image deployment/nginx nginx=nginx:1.28
kubectl delete deployment nginx

Declarative configuration tells Kubernetes what the end state should look like. You write a manifest (a YAML file), and kubectl apply sends it to the API server. Kubernetes computes the difference between what exists and what you declared, then makes only the necessary changes.

# deployment.yaml \u2014 the desired state
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.28

# Declarative: apply the manifest \u2014 works for create AND update
kubectl apply -f deployment.yaml

The Reconciliation Loop: Observe \u2192 Diff \u2192 Act

Declaring desired state is only half the story. The other half is reconciliation \u2014 the continuous process by which Kubernetes controllers drive actual state toward desired state. Every controller in Kubernetes follows the same three-phase pattern.

1. Observe \u2014 Read the current state of the world from the API server (the objects the controller is responsible for, plus any dependent resources).
2. Diff \u2014 Compare the observed state against the desired state declared in the object\u2019s spec.
3. Act \u2014 If there is a gap, take the minimum set of actions to close it: create missing resources, update drifted ones, or delete ones that should no longer exist. Then update the object\u2019s status subresource to reflect the new reality.

This loop runs continuously. It is not a one-shot process triggered by your kubectl apply. After you apply a Deployment with replicas: 3, the Deployment controller does not just create three pods and walk away. It keeps watching. If a pod dies at 3 AM, the controller notices and creates a replacement \u2014 no human intervention required.

stateDiagram-v2
    [*] --> DesiredStateDeclared: User applies manifest
    DesiredStateDeclared --> Watching: Controller starts watch
    Watching --> DriftDetected: Actual \u2260 Desired
    Watching --> Watching: Actual = Desired (no-op)
    DriftDetected --> Acting: Create / Update / Delete resources
    Acting --> StatusUpdate: Write status to API server
    StatusUpdate --> Watching: Resume watch loop
    

Watches: Event-Driven, Not Polling

A naive implementation of the reconciliation loop would poll the API server on a timer \u2014 "check every 5 seconds if anything changed." Kubernetes is smarter than that. Controllers use the API server\u2019s watch mechanism, a long-lived HTTP connection (using chunked transfer encoding) that streams change events in real time.

When a controller starts, it performs a list operation to get the full current state, then opens a watch starting from the resource version returned by the list. The API server pushes ADDED, MODIFIED, and DELETED events as they happen. This design means controllers react to changes within milliseconds, not on some polling interval.

# You can see the watch mechanism in action with kubectl
# This opens a long-lived connection and streams events as pods change
kubectl get pods --watch

# Raw API equivalent \u2014 note the watch=true query parameter
kubectl get pods --v=8 2>&1 | grep "GET"
# Output shows: GET https://<api-server>/api/v1/namespaces/default/pods?watch=true

Internally, the client-go library (used by all built-in controllers and most custom ones) wraps this list+watch pattern in a component called an Informer. An Informer maintains a local in-memory cache of the resources it watches, so controllers can read state without hitting the API server on every reconciliation. It also de-duplicates events and feeds them into a work queue, ensuring each resource is reconciled at most once at a time.

Eventual Consistency: The Kubernetes Bargain

Kubernetes is an eventually consistent system. When you kubectl apply a manifest, the API server acknowledges the write to etcd and returns immediately. But the actual resources \u2014 the pods, the endpoints, the iptables rules \u2014 are not yet created. That work happens asynchronously as each controller picks up the change and acts on it.

This means there is always a window of time where actual state does not match desired state. A Deployment can exist in etcd before its pods are running. A Service can be created before its endpoints are populated. The gap is normally small (seconds), but under heavy load or during node failures, convergence can take longer.

This is a deliberate tradeoff. Kubernetes optimizes for resilience and scalability over immediate consistency. In a system managing thousands of nodes and tens of thousands of pods, attempting synchronous, transactional updates across all components would be impossibly slow and fragile. Instead, each controller independently converges its slice of the world, and the system as a whole settles into the desired state.

Drift Detection and Self-Healing in Practice

The real power of the reconciliation model becomes visible when things go wrong. The following scenarios demonstrate how Kubernetes detects drift between desired and actual state, then automatically heals.

Scenario 1: Pod Deletion \u2014 The Controller Recreates It

Start with a Deployment running three replicas. Manually delete one pod. The ReplicaSet controller notices the count dropped to 2, compares it against the desired count of 3, and immediately creates a replacement.

# Create a deployment with 3 replicas
kubectl apply -f deployment.yaml

# Confirm 3 pods are running
kubectl get pods -l app=nginx
# NAME                     READY   STATUS    AGE
# nginx-7c5ddbdf54-abc12   1/1     Running   30s
# nginx-7c5ddbdf54-def34   1/1     Running   30s
# nginx-7c5ddbdf54-ghi56   1/1     Running   30s

# Delete one pod manually \u2014 simulating a crash
kubectl delete pod nginx-7c5ddbdf54-abc12

# Within seconds, a replacement appears
kubectl get pods -l app=nginx
# NAME                     READY   STATUS    AGE
# nginx-7c5ddbdf54-def34   1/1     Running   60s
# nginx-7c5ddbdf54-ghi56   1/1     Running   60s
# nginx-7c5ddbdf54-xyz99   1/1     Running   3s    <-- new pod

Scenario 2: Imperative Drift \u2014 Declarative Correction

What happens if someone uses kubectl scale to manually change the replica count on a live Deployment? The cluster state drifts from what the manifest declares. The next kubectl apply of the original manifest detects the difference and corrects it. This is why imperative edits on declaratively managed resources are dangerous \u2014 the next apply will overwrite them.

# Someone scales the deployment imperatively
kubectl scale deployment/nginx --replicas=5

# The cluster now has 5 pods. But the manifest says 3.
# Re-apply the manifest to restore desired state:
kubectl apply -f deployment.yaml

# The controller terminates the 2 extra pods
kubectl get pods -l app=nginx
# Back to 3 pods

Scenario 3: Node Failure \u2014 Rescheduling to Healthy Nodes

When a node becomes unreachable, the node controller marks it as NotReady after a timeout (default: 40 seconds). The pod eviction controller then taints the node, and pods bound to it begin terminating. The ReplicaSet controller sees replica counts drop below the desired number and schedules replacements on healthy nodes \u2014 all without human intervention.

# Monitor the self-healing process during a node failure
kubectl get nodes --watch
# NAME     STATUS     ROLES    AGE   VERSION
# node-1   Ready      <none>   10d   v1.30.2
# node-2   NotReady   <none>   10d   v1.30.2   <-- node went down

kubectl get pods -l app=nginx -o wide --watch
# Pods on node-2 enter Terminating status
# New pods are scheduled on node-1 (or other healthy nodes)

# Check events to see the reconciliation in action
kubectl get events --sort-by=.lastTimestamp | tail -5
# 2m   Normal   SuccessfulCreate   replicaset/nginx-7c5ddbdf54   Created pod: nginx-7c5ddbdf54-new01

Multiple Controllers, One Convergence

A single kubectl apply of a Deployment triggers a cascade of controllers, each reconciling its own piece of the puzzle. No single controller handles the full lifecycle \u2014 they compose together through the objects they create and own.

1. The Deployment controller sees the new Deployment object. It creates (or updates) a ReplicaSet to match the pod template.
2. The ReplicaSet controller sees the new ReplicaSet. It counts existing pods matching the selector. If the count is too low, it creates new Pod objects.
3. The Scheduler sees unscheduled Pods (those with no nodeName). It assigns each pod to a node by writing the nodeName field.
4. The kubelet on each assigned node watches for pods bound to it. It pulls the container image and starts the container through the container runtime.
5. The Endpoint/EndpointSlice controller sees pods become Ready. It adds their IPs to the corresponding Service\u2019s endpoint list.
6. kube-proxy (or the CNI plugin) picks up the new endpoints and updates the node\u2019s network rules so traffic can reach the new pods.

Each controller independently converges its own resources. The Deployment controller does not know or care about iptables rules. The scheduler does not know about Services. Yet the final result \u2014 a fully networked, load-balanced set of running containers \u2014 emerges from their independent reconciliation loops.

Why This Matters

The declarative model with reconciliation loops gives you three properties that are hard to achieve with imperative scripting:

• Self-healing. The system continuously repairs itself. Crashed pods are restarted, failed nodes are drained, and missing resources are recreated. You do not need monitoring scripts or cron jobs to do this \u2014 it is built into the architecture.
• Idempotent operations. Running kubectl apply -f manifest.yaml ten times has the same result as running it once. This makes automation safe and CI/CD pipelines reliable. You never have to worry about "is this resource already created?"
• Git as the source of truth. Because the desired state is a YAML file, it lives in version control. You get history, code review, rollback, and audit trails for free. This is the foundation of the GitOps workflow pattern covered later in this guide.

Pods — The Smallest Deployable Unit

A Pod is the smallest object you can create in Kubernetes. It represents a group of one or more containers that share a network identity, an IPC namespace, and optionally a set of storage volumes. When Kubernetes schedules work onto a node, it schedules entire Pods — never individual containers.

Most Pods you encounter in the wild contain a single container. But the abstraction exists as a group because some workloads genuinely need tightly coupled processes: a web server and a log-shipping sidecar, or an application container alongside a service-mesh proxy. These containers are co-located, co-scheduled, and run in a shared context.

What Containers in a Pod Share

Containers inside the same Pod are not isolated from each other the way separate Pods are. They share three key namespaces, which has concrete implications for how you design your applications.

Each Pod gets its own unique cluster IP address. Other Pods in the cluster communicate with it using that IP, regardless of how many containers are running inside. From the network’s perspective, a Pod is a single host.

Pod Lifecycle

Every Pod moves through a defined set of phases from creation to termination. Understanding these phases is critical for debugging — when a Pod is stuck, its phase tells you where in the lifecycle it stalled.

stateDiagram-v2
    [*] --> Pending
    Pending --> Running : Scheduled, images pulled, containers starting
    Running --> Succeeded : All containers exit with code 0
    Running --> Failed : A container exits with non-zero code
    Pending --> Failed : Image pull fails or cannot be scheduled
    Running --> Unknown : Node becomes unreachable
    Unknown --> Running : Node reconnects
    Unknown --> Failed : Node stays unreachable beyond timeout
    Succeeded --> [*]
    Failed --> [*]
    

Pod Phases

Pod Conditions

While the phase gives you a high-level summary, conditions provide a more granular picture. Each condition is a boolean with a reason and a timestamp. You can inspect them with kubectl describe pod or query them in JSON output.

Container States

Each container inside a Pod has its own state, independent of the Pod phase. Kubernetes tracks three possible states per container:

• Waiting — The container is not yet running. The reason field tells you why: ContainerCreating, ImagePullBackOff, CrashLoopBackOff, etc.
• Running — The container is executing. The startedAt timestamp tells you when it began.
• Terminated — The container finished execution. You get the exitCode, reason, and both startedAt and finishedAt timestamps.

Restart Policies

The restartPolicy field in the Pod spec controls what the kubelet does when a container exits. It applies to all containers in the Pod — you cannot set different restart policies for different containers. The default is Always.

When a container repeatedly crashes, the kubelet delays restarts using exponential backoff. This is the infamous CrashLoopBackOff status — it means the kubelet is waiting before retrying. The delay caps at 5 minutes.

Anatomy of a Pod Spec

A Pod spec is more than just a list of containers. It includes scheduling constraints, security settings, volume definitions, and metadata that controllers and the scheduler use to make decisions. Here is an annotated example that shows the most commonly used fields.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app
    version: v1
spec:
  serviceAccountName: my-app-sa        # Identity for RBAC and API access
  restartPolicy: Always

  # --- Scheduling constraints ---
  nodeSelector:
    disk: ssd                           # Only nodes with this label
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"             # Tolerate the gpu taint
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app: my-app
            topologyKey: kubernetes.io/hostname  # Spread across nodes

  # --- Security ---
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000

  # --- Volumes ---
  volumes:
    - name: config-vol
      configMap:
        name: my-app-config
    - name: data-vol
      persistentVolumeClaim:
        claimName: my-app-data
    - name: tmp
      emptyDir: {}

  # --- Containers ---
  containers:
    - name: app
      image: my-app:1.2.0
      ports:
        - containerPort: 8080
      env:
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: host
      volumeMounts:
        - name: config-vol
          mountPath: /etc/app/config
          readOnly: true
        - name: data-vol
          mountPath: /var/data
        - name: tmp
          mountPath: /tmp
      resources:
        requests:
          cpu: "250m"
          memory: "128Mi"
        limits:
          cpu: "500m"
          memory: "256Mi"
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 10
        periodSeconds: 15
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5

A few things to note in this spec. The serviceAccountName determines what Kubernetes API permissions the Pod has — always set it explicitly rather than relying on the default service account. The securityContext at the Pod level applies to all containers; you can override it per-container for finer control. Scheduling fields like nodeSelector, tolerations, and affinity work together to control where the Pod lands.

Creating and Inspecting Pods

You will rarely create bare Pods in production — Deployments and Jobs handle that for you. But during development and debugging, working directly with Pods is essential. Here are the most common operations.

Create a Pod Imperatively

The fastest way to spin up a Pod for quick testing. The --rm and -it flags make it behave like a temporary interactive shell.

# Run a one-off Pod (deleted automatically when you exit)
kubectl run tmp-shell --rm -it --image=alpine -- /bin/sh

# Run a Pod with a specific command
kubectl run dns-test --image=busybox --restart=Never -- nslookup kubernetes.default

# Generate a Pod YAML without creating it (dry-run)
kubectl run my-app --image=nginx:1.25 --port=80 --dry-run=client -o yaml > pod.yaml

Create a Pod Declaratively

# Apply a Pod manifest
kubectl apply -f pod.yaml

# Watch the Pod come up
kubectl get pod my-app -w

Inspecting Pod State

kubectl get shows a summary. kubectl describe gives you the full picture: events, conditions, container states, and reasons for failures. When something goes wrong, describe is always your first stop.

# Summary with status, restarts, age, IP, and node
kubectl get pod my-app -o wide

# Full details: events, conditions, container states
kubectl describe pod my-app

# Logs from the primary container
kubectl logs my-app

# Logs from a specific container in a multi-container Pod
kubectl logs my-app -c sidecar

# Stream logs in real time
kubectl logs my-app -f --tail=50

# Check container states as JSON
kubectl get pod my-app -o jsonpath='{.status.containerStatuses[*].state}'

Debugging Pods

When a container is running but misbehaving, kubectl exec lets you open a shell inside it. This works only if the container has a shell binary — distroless and minimal images often do not. That is where ephemeral debug containers come in.

Exec into a Running Container

# Open an interactive shell
kubectl exec -it my-app -- /bin/sh

# Run a one-off command
kubectl exec my-app -- cat /etc/app/config/settings.yaml

# Exec into a specific container in a multi-container Pod
kubectl exec -it my-app -c sidecar -- /bin/bash

Ephemeral Debug Containers

Introduced as stable in Kubernetes 1.25, ephemeral containers solve the “distroless debugging” problem. You inject a temporary container — with the tools you need — into a running Pod. The debug container shares the Pod’s namespaces, so it can see the same network interfaces, processes, and file systems.

# Attach a debug container that shares the target container's PID namespace
kubectl debug -it my-app --image=busybox --target=app

# Debug a CrashLoopBackOff Pod by copying it with a different command
kubectl debug my-app -it --copy-to=my-app-debug --container=app -- /bin/sh

# Debug at the node level (creates a privileged Pod on the node)
kubectl debug node/worker-1 -it --image=ubuntu

Deleting Pods

When you delete a Pod, Kubernetes sends a SIGTERM to every container and waits up to the terminationGracePeriodSeconds (default: 30 seconds) for a clean shutdown. If containers are still running after that deadline, it sends SIGKILL.

# Graceful delete (waits for terminationGracePeriodSeconds)
kubectl delete pod my-app

# Force delete (skip the grace period — use sparingly)
kubectl delete pod my-app --grace-period=0 --force

# Delete all Pods matching a label
kubectl delete pods -l app=my-app

# Delete a Pod from a YAML file
kubectl delete -f pod.yaml

ReplicaSets and Deployments — Managing Pod Replicas

In the previous section, you learned that a Pod is the smallest deployable unit in Kubernetes. But here is the thing: you almost never create Pods directly. A bare Pod has no self-healing capability — if it crashes, gets evicted, or its node goes down, it is gone forever. Nothing recreates it.

This is where ReplicaSets and Deployments come in. They form a two-layer abstraction that keeps your application running at the desired scale and gives you controlled rollout and rollback capabilities. Understanding how these layers interact is essential to operating anything in production on Kubernetes.

ReplicaSets: The Replication Engine

A ReplicaSet has one job: ensure that a specified number of identical Pod replicas are running at all times. It does this through a continuous reconciliation loop. The ReplicaSet controller watches the cluster state, compares the current number of matching Pods to the desired count in the spec, and creates or deletes Pods to close the gap.

The "matching" part is critical. A ReplicaSet finds its Pods using a label selector, not by tracking specific Pod names. It looks for Pods whose labels match its spec.selector, counts them, and acts accordingly. This decoupled relationship means a ReplicaSet can even adopt pre-existing Pods if their labels match.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: web-frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-frontend
  template:
    metadata:
      labels:
        app: web-frontend
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          ports:
            - containerPort: 80

Three key fields define a ReplicaSet: spec.replicas sets the desired Pod count, spec.selector defines which labels to match, and spec.template provides the Pod blueprint for creating new replicas. The labels in template.metadata.labels must match the selector — Kubernetes validates this and rejects the object if they do not align.

Deployments: The Abstraction You Actually Use

A Deployment is a higher-level controller that manages ReplicaSets on your behalf. When you create a Deployment, it creates a ReplicaSet. When you update the Deployment's Pod template (for example, changing the container image), the Deployment creates a new ReplicaSet with the updated template and gradually scales it up while scaling the old one down. This is how rolling updates work.

The ownership chain is: Deployment → ReplicaSet → Pods. The Deployment never manages Pods directly. It manages ReplicaSets, and each ReplicaSet manages its own set of Pods. Old ReplicaSets are kept around (scaled to zero) so that rollbacks can reactivate them instantly.

graph TD
    D["Deployment<br/><strong>web-app</strong>"]
    RS1["ReplicaSet · revision 1<br/><em>nginx:1.24</em><br/>replicas: 0"]
    RS2["ReplicaSet · revision 2<br/><em>nginx:1.25</em><br/>replicas: 3"]
    P1["Pod web-app-a7x2k"]
    P2["Pod web-app-b9m3p"]
    P3["Pod web-app-c4n8q"]

    D --> RS1
    D --> RS2
    RS2 --> P1
    RS2 --> P2
    RS2 --> P3

    style D fill:#326ce5,stroke:#fff,color:#fff
    style RS1 fill:#666,stroke:#999,color:#ccc
    style RS2 fill:#1a73e8,stroke:#fff,color:#fff
    style P1 fill:#4caf50,stroke:#fff,color:#fff
    style P2 fill:#4caf50,stroke:#fff,color:#fff
    style P3 fill:#4caf50,stroke:#fff,color:#fff
    

Deployment Spec Anatomy

A Deployment spec is structurally similar to a ReplicaSet — it has replicas, selector, and template — but adds fields for update strategy, revision history, and rollout behavior. Here is a production-ready Deployment with every important field annotated.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  labels:
    app: web-app
spec:
  replicas: 3
  revisionHistoryLimit: 5          # Keep 5 old ReplicaSets for rollback
  selector:
    matchLabels:
      app: web-app
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                  # At most 1 extra Pod during update
      maxUnavailable: 0            # Never drop below desired count
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: web-app
          image: myregistry/web-app:v2.4.1
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

Deployment Strategies

Kubernetes offers two built-in strategies for how a Deployment replaces old Pods with new ones. Choosing the right one depends on whether your application can tolerate running two versions simultaneously.

RollingUpdate (Default)

The RollingUpdate strategy incrementally replaces Pods. Kubernetes creates new Pods from the updated ReplicaSet and terminates old Pods from the previous ReplicaSet in a controlled sequence. At no point does the total count of available Pods drop to zero (assuming sane settings), which makes this the standard choice for stateless web services.

Two parameters control the pace of the rollout:

Setting maxSurge: 1 and maxUnavailable: 0 is the most conservative configuration — Kubernetes spins up one new Pod, waits for it to become ready, then terminates one old Pod, and repeats. This is slower but guarantees zero capacity loss. Setting both to higher values (or percentages) speeds up the rollout at the cost of temporary over-provisioning or reduced capacity.

Recreate

The Recreate strategy is simpler and more brutal: it kills all existing Pods before creating any new ones. This means a period of complete downtime during the update. Use this only when your application cannot run two versions side-by-side — for example, when a new version requires an exclusive database migration lock or uses an incompatible on-disk format.

spec:
  strategy:
    type: Recreate     # No rollingUpdate block needed

Rollout Management

Every change to a Deployment’s .spec.template triggers a new rollout. Kubernetes tracks these rollouts as revisions, and you can inspect, pause, and undo them using kubectl rollout commands. Changes to fields outside the template (like replicas) do not trigger a new rollout.

Watching a Rollout in Progress

# Trigger a rollout by updating the image
kubectl set image deployment/web-app web-app=myregistry/web-app:v2.5.0

# Watch the rollout progress in real-time
kubectl rollout status deployment/web-app
# Waiting for deployment "web-app" rollout to finish:
#   1 out of 3 new replicas have been updated...
#   2 out of 3 new replicas have been updated...
#   3 out of 3 new replicas have been updated...
#   deployment "web-app" successfully rolled out

Viewing Rollout History

Each rollout creates a new revision. You can inspect the full history and see what changed in each revision. Adding the --record flag (or using annotations) when making changes captures the command that triggered each revision.

# List all revisions
kubectl rollout history deployment/web-app
# REVISION  CHANGE-CAUSE
# 1         kubectl apply --filename=web-app.yaml
# 2         kubectl set image deployment/web-app web-app=myregistry/web-app:v2.4.1
# 3         kubectl set image deployment/web-app web-app=myregistry/web-app:v2.5.0

# Inspect a specific revision’s Pod template
kubectl rollout history deployment/web-app --revision=2

Rolling Back

When a new release is broken, you can instantly revert to a previous revision. The rollback does not re-deploy the old image from scratch — it reactivates the old ReplicaSet (which was kept around at zero replicas) and scales it back up. This makes rollbacks fast.

# Roll back to the previous revision
kubectl rollout undo deployment/web-app

# Roll back to a specific revision
kubectl rollout undo deployment/web-app --to-revision=2

# Verify the rollback completed
kubectl rollout status deployment/web-app

Revision History Limits

Every old ReplicaSet consumes a small amount of etcd storage and clutters kubectl get rs output. The spec.revisionHistoryLimit field controls how many old ReplicaSets are retained. The default is 10. Setting it to 0 disables rollback entirely because no old ReplicaSets are preserved. A value between 3 and 10 is practical for most workloads.

Scaling

Scaling a Deployment changes the replicas field on the active ReplicaSet. You can do this imperatively with kubectl scale or declaratively by updating the manifest. Since scaling does not change the Pod template, it does not trigger a new rollout or create a new revision.

# Imperative scaling
kubectl scale deployment/web-app --replicas=5

# Verify the new replica count
kubectl get deployment web-app
# NAME      READY   UP-TO-DATE   AVAILABLE   AGE
# web-app   5/5     5            5           12d

For declarative scaling, update spec.replicas in your YAML and apply it. For automatic scaling based on CPU or memory utilization, use a HorizontalPodAutoscaler (HPA) — which adjusts the replica count on the Deployment dynamically. When using an HPA, you typically omit spec.replicas from your manifest to avoid conflicts between the HPA controller and your declared value.

# Create an HPA targeting 70% CPU utilization, scaling between 3 and 10 replicas
kubectl autoscale deployment/web-app --min=3 --max=10 --cpu-percent=70

Real-World Patterns: Blue-Green and Canary Deployments

The built-in RollingUpdate strategy works well for most scenarios, but some teams need finer control over traffic shifting during deploys. Kubernetes does not have first-class "blue-green" or "canary" resources, but you can implement both patterns using native primitives: Deployments, Services, and label selectors.

Blue-Green Deployments

In a blue-green deployment, you run two complete environments side by side — "blue" (current) and "green" (new). All traffic goes to one environment at a time. Once the green environment is validated, you switch traffic instantly by updating the Service selector. If something goes wrong, you switch back.

# deployment-blue.yaml — the currently active version
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
      version: blue
  template:
    metadata:
      labels:
        app: web-app
        version: blue
    spec:
      containers:
        - name: web-app
          image: myregistry/web-app:v2.4.1
---
# deployment-green.yaml — the new version, deployed alongside blue
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
      version: green
  template:
    metadata:
      labels:
        app: web-app
        version: green
    spec:
      containers:
        - name: web-app
          image: myregistry/web-app:v2.5.0
---
# service.yaml — traffic switch controlled by the 'version' label
apiVersion: v1
kind: Service
metadata:
  name: web-app
spec:
  selector:
    app: web-app
    version: blue      # ← Change to "green" to switch traffic
  ports:
    - port: 80
      targetPort: 8080

The cutover is a single operation: patch the Service selector from blue to green. Traffic shifts immediately because the Service updates its endpoint list. To roll back, patch the selector back to blue. Once you are confident the green version is stable, delete the blue Deployment.

# Switch traffic from blue to green
kubectl patch svc web-app -p '{"spec":{"selector":{"version":"green"}}}'

# Rollback: switch traffic back to blue
kubectl patch svc web-app -p '{"spec":{"selector":{"version":"blue"}}}'

Canary Deployments

A canary deployment routes a small percentage of traffic to the new version while the majority continues hitting the stable version. In native Kubernetes, you achieve this by running two Deployments with a shared label that the Service selects on. Traffic is distributed across all matching Pods proportionally.

# Stable deployment — 9 replicas serving ~90% of traffic
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: web-app
      track: stable
  template:
    metadata:
      labels:
        app: web-app        # ← Shared label the Service selects on
        track: stable
    spec:
      containers:
        - name: web-app
          image: myregistry/web-app:v2.4.1
---
# Canary deployment — 1 replica serving ~10% of traffic
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-app
      track: canary
  template:
    metadata:
      labels:
        app: web-app        # ← Same shared label
        track: canary
    spec:
      containers:
        - name: web-app
          image: myregistry/web-app:v2.5.0
---
# Service selects ONLY on app: web-app — matches both Deployments
apiVersion: v1
kind: Service
metadata:
  name: web-app
spec:
  selector:
    app: web-app            # ← Does NOT include 'track', so both match
  ports:
    - port: 80
      targetPort: 8080

Since the Service selects on app: web-app only, it targets all 10 Pods (9 stable + 1 canary). Kubernetes distributes requests roughly evenly across endpoints, so about 10% of traffic hits the canary. You control the ratio by adjusting replica counts. To promote the canary, update the stable Deployment’s image and scale the canary back to zero.

Putting It All Together

Here is a summary of the relationship between the resources covered in this section and when to reach for each pattern:

In the next section, you will learn about StatefulSets — the controller designed for applications that need stable network identities and persistent storage, where the interchangeable nature of Deployment-managed Pods breaks down.

StatefulSets — Stable Identity for Stateful Applications

Not every workload is stateless. Databases need persistent disk, message queues need stable network addresses, and distributed stores need to know which replica they are. A Deployment treats every Pod as interchangeable — when a Pod dies, it gets a new random name, a new IP, and its local storage vanishes. That model breaks fundamentally for anything that stores data or participates in a cluster protocol.

StatefulSets solve this by giving each Pod a persistent identity that survives rescheduling. The Pod name, its DNS hostname, and its storage volume are all stable. Pod mysql-0 is always mysql-0, whether it runs on node A today or node B tomorrow.

Why Deployments Fall Short for Stateful Workloads

Consider a 3-node PostgreSQL cluster running in streaming replication. The primary (node 0) writes the WAL, and replicas (nodes 1 and 2) connect to the primary by hostname to stream changes. If you use a Deployment, you immediately hit three problems: Pod names are random (so replicas cannot find the primary), storage is ephemeral (so a restarted Pod loses all data), and Pods are created and destroyed in any order (so the primary might not exist when replicas start).

The Three Guarantees of StatefulSets

1. Stable Network Identity

Each Pod in a StatefulSet gets a predictable hostname following the pattern <statefulset-name>-<ordinal>. A StatefulSet named postgres with 3 replicas creates Pods named postgres-0, postgres-1, and postgres-2. When postgres-1 is rescheduled to a different node, it keeps the name postgres-1 — and critically, its DNS record points to the new IP.

This identity is paired with a headless Service (covered below) to give each Pod a stable DNS name like postgres-0.postgres-headless.default.svc.cluster.local. Other components can connect to a specific replica by name, which is exactly what replication protocols require.

2. Stable Persistent Storage

StatefulSets use volumeClaimTemplates to create a dedicated PersistentVolumeClaim for each Pod. When the StatefulSet creates postgres-0, it also creates a PVC named data-postgres-0. If postgres-0 is deleted and recreated, the new Pod reattaches to the same data-postgres-0 PVC — your data survives.

3. Ordered Deployment and Scaling

By default, StatefulSets create Pods sequentially. Pod 0 must be Running and Ready before Pod 1 starts. This ordering matters for leader-election or primary-replica setups where the first node must initialize the cluster before replicas join. Scaling down reverses the order: the highest-ordinal Pod is terminated first.

flowchart LR
    subgraph "StatefulSet: postgres (replicas=3)"
        direction LR
        P0["postgres-0<br/><small>Created first</small>"] -->|"Ready ✓"| P1["postgres-1<br/><small>Created second</small>"]
        P1 -->|"Ready ✓"| P2["postgres-2<br/><small>Created third</small>"]
    end

    subgraph "PersistentVolumeClaims"
        PVC0["data-postgres-0<br/>10Gi"]
        PVC1["data-postgres-1<br/>10Gi"]
        PVC2["data-postgres-2<br/>10Gi"]
    end

    P0 -.->|"bound"| PVC0
    P1 -.->|"bound"| PVC1
    P2 -.->|"bound"| PVC2
    

Headless Services — Direct Pod DNS

A normal ClusterIP Service gives you a single virtual IP that load-balances across all Pods. That is useless when you need to connect to a specific Pod — you cannot tell a replica "connect to the primary at this VIP" because the VIP might route to any backend Pod.

A headless Service is a Service with clusterIP: None. Instead of creating a single VIP, it creates individual DNS A records for each Pod. This gives every StatefulSet Pod a predictable, resolvable hostname.

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
  labels:
    app: postgres
spec:
  clusterIP: None          # This makes it headless
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432

With this Service in place, each Pod gets a DNS entry following the pattern:

# Pattern:
# <pod-name>.<headless-service>.<namespace>.svc.cluster.local

postgres-0.postgres-headless.default.svc.cluster.local  → 10.244.1.5
postgres-1.postgres-headless.default.svc.cluster.local  → 10.244.2.8
postgres-2.postgres-headless.default.svc.cluster.local  → 10.244.3.3

The StatefulSet's spec.serviceName field must reference this headless Service. This is how Kubernetes knows which Service to register Pod DNS records with. Without this link, your Pods will not get individual DNS entries.

Anatomy of a StatefulSet Manifest

A StatefulSet spec looks similar to a Deployment, with two important additions: the serviceName field that links to the headless Service, and the volumeClaimTemplates section that defines per-Pod storage. Here is the complete structure for a 3-replica PostgreSQL cluster:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless   # Must match the headless Service name
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 1Gi
  volumeClaimTemplates:            # One PVC per Pod
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: standard
        resources:
          requests:
            storage: 10Gi

The volumeClaimTemplates section acts as a PVC template. For each Pod ordinal, Kubernetes creates a PVC named <template-name>-<statefulset-name>-<ordinal> — in this case data-postgres-0, data-postgres-1, and data-postgres-2. These PVCs are not deleted when Pods are rescheduled, which is exactly what you want for a database.

Practical Example: PostgreSQL with Streaming Replication

A real-world replicated PostgreSQL setup needs each Pod to know its role (primary vs. replica) and configure itself accordingly. The ordinal index makes this straightforward: Pod 0 initializes as primary, all others connect to Pod 0 as replicas.

Start by creating the Secret and headless Service:

# Create the password secret
kubectl create secret generic postgres-secret \
  --from-literal=password='S3cur3P@ss' \
  --from-literal=replication-password='R3plP@ss'

# Apply the headless Service
kubectl apply -f postgres-headless-svc.yaml

Next, use a ConfigMap to hold an init script that detects whether the Pod is the primary or a replica based on its hostname ordinal:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-init
data:
  setup.sh: |
    #!/bin/bash
    set -e

    # Extract ordinal from hostname (e.g., "postgres-0" -> "0")
    ORDINAL="${HOSTNAME##*-}"

    if [ "$ORDINAL" -eq 0 ]; then
      echo "Initializing as PRIMARY"
      # Primary-specific config: enable replication slots, WAL shipping
      cat >> /var/lib/postgresql/data/pgdata/postgresql.conf <<CONF
    wal_level = replica
    max_wal_senders = 5
    max_replication_slots = 5
    CONF
    else
      echo "Initializing as REPLICA — connecting to postgres-0"
      # Use pg_basebackup to clone data from the primary
      pg_basebackup -h postgres-0.postgres-headless \
        -U replicator -D /var/lib/postgresql/data/pgdata \
        -Fp -Xs -R
    fi

The key insight is the HOSTNAME environment variable. Kubernetes sets it to the Pod name (postgres-0, postgres-1, etc.), so you can parse the ordinal and branch your initialization logic. Replicas connect to postgres-0.postgres-headless — the stable DNS name — regardless of what node the primary is running on.

Pod Management Policy

The default podManagementPolicy is OrderedReady, which enforces the sequential startup and shutdown behavior described above. For workloads that do not need ordering — for example, a distributed cache where all nodes are peers — you can set it to Parallel to launch all Pods at once.

spec:
  podManagementPolicy: Parallel    # All Pods start/stop simultaneously
  replicas: 5
  serviceName: memcached-headless

Update Strategies

StatefulSets support two update strategies that control how Pods are replaced when you change the Pod template (e.g., updating the container image).

RollingUpdate (default)

Pods are updated one at a time in reverse ordinal order (highest ordinal first). This is intentional — in most primary/replica setups, replicas have higher ordinals and should be updated before the primary. Each Pod must become Running and Ready before the next one is updated.

RollingUpdate with Partition (Canary Deploys)

The partition parameter lets you perform a staged rollout. Only Pods with an ordinal greater than or equal to the partition value are updated. This is a powerful canary mechanism — you can test a new image on the last replica before rolling it out to the entire set.

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2             # Only pods with ordinal >= 2 are updated

# Step 1: Set partition to 2, update the image
# Only postgres-2 gets the new image
kubectl patch statefulset postgres -p \
  '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
kubectl set image statefulset/postgres postgres=postgres:17

# Step 2: Verify postgres-2 is healthy with the new version
kubectl get pod postgres-2 -o jsonpath='{.spec.containers[0].image}'

# Step 3: Lower partition to 0 to roll out to all Pods
kubectl patch statefulset postgres -p \
  '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'

OnDelete

With OnDelete, Kubernetes does not automatically update Pods when you change the template. Instead, you manually delete Pods one by one, and each replacement is created with the new template. This gives you full control over the update pace and order, which can be critical for databases where you must manually verify replication health between each step.

spec:
  updateStrategy:
    type: OnDelete

Common Pitfalls

PVC Retention: Volumes Outlive the StatefulSet

When you delete a StatefulSet or scale it down, the PVCs are not automatically deleted. This is a safety feature — you do not want to lose a database volume because someone ran kubectl delete statefulset. However, it means orphaned PVCs accumulate and continue to consume storage (and cost money) unless you clean them up.

# List PVCs that belonged to a deleted StatefulSet
kubectl get pvc -l app=postgres

# Manually delete orphaned PVCs after confirming data is backed up
kubectl delete pvc data-postgres-0 data-postgres-1 data-postgres-2

Starting with Kubernetes 1.27 (stable), you can configure automatic PVC cleanup using persistentVolumeClaimRetentionPolicy:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete        # Delete PVCs when StatefulSet is deleted
    whenScaled: Retain         # Keep PVCs when scaling down (safe default)

Ordering Dependencies and Stuck Rollouts

With OrderedReady, if Pod 0 fails to become Ready, the entire StatefulSet is stuck — Pods 1 and 2 will never be created. This cascading failure is one of the most common StatefulSet debugging scenarios. Always check the Pod 0 logs and events first when a StatefulSet is not scaling up.

# Diagnose a stuck StatefulSet
kubectl describe statefulset postgres
kubectl get pods -l app=postgres
kubectl logs postgres-0 --previous    # Check crash logs
kubectl describe pod postgres-0       # Check events for scheduling/volume issues

Scaling Down Safely

Scaling down removes Pods in reverse ordinal order (highest first), which is usually the safest order for primary-replica setups. However, Kubernetes does not understand your application's replication state. If postgres-2 holds data that has not been replicated elsewhere, scaling from 3 to 2 can cause data loss. Always verify your application's replication health before scaling down.

When to Use Operators Instead

Raw StatefulSets give you identity, storage, and ordering — but they do not understand your application. They cannot perform automated failover when a PostgreSQL primary dies, rebalance partitions in a Kafka cluster, or trigger a backup before scaling down. For production databases and complex distributed systems, a Kubernetes Operator wraps the StatefulSet with application-specific automation.

DaemonSets, Jobs, and CronJobs — Specialized Workload Controllers

Deployments and StatefulSets cover most workloads, but not all work fits the “run N replicas forever” model. Some pods need to run on every node — log shippers, monitoring agents, network plugins. Others need to run once, finish, and exit. Still others need to fire on a schedule, like a nightly database backup. Kubernetes provides three specialized controllers for exactly these patterns: DaemonSets, Jobs, and CronJobs.

Each controller builds on the same reconciliation loop that powers Deployments, but they differ in when pods are created, where they are placed, and what happens when a pod finishes. Understanding these differences lets you model your entire workload landscape without resorting to external cron daemons, systemd services, or manual node-by-node deployments.

DaemonSets — One Pod Per Node

A DaemonSet ensures that every node (or a targeted subset) runs exactly one copy of a pod. When a new node joins the cluster, the DaemonSet controller automatically schedules a pod on it. When a node is removed, the pod is garbage-collected. You never specify a replica count — the node count is the replica count.

This makes DaemonSets the natural choice for node-level infrastructure that must be present everywhere in the cluster. The most common use cases include:

• Log collectors — Fluentd, Fluent Bit, or Logstash agents that tail container logs from /var/log and forward them to a central store.
• Monitoring agents — Prometheus node-exporter for hardware and OS metrics, Datadog agents, or New Relic infrastructure agents.
• CNI / network plugins — Calico, Cilium, and AWS VPC CNI all run as DaemonSets to configure networking on each node.
• Storage daemons — Ceph OSDs, Longhorn engine processes, or CSI node plugins that need access to the host’s disk and device tree.

Basic DaemonSet YAML

The following manifest deploys Fluent Bit as a log collector across every node. Notice there is no replicas field — the DaemonSet controller handles pod placement automatically.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:3.1
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: containers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: containers
          hostPath:
            path: /var/lib/docker/containers

Targeting a Subset of Nodes

You don’t always want a pod on every node. Maybe your GPU monitoring agent should only run on nodes with GPUs, or a storage daemon should only run on nodes with SSDs. Use nodeSelector or nodeAffinity inside the pod template to restrict placement. The DaemonSet controller will only schedule pods on nodes that match.

# Inside spec.template.spec:
nodeSelector:
  node.kubernetes.io/instance-type: gpu

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

Tolerations are equally important. Control-plane nodes are typically tainted with node-role.kubernetes.io/control-plane:NoSchedule. If you need your DaemonSet to run on control-plane nodes too (for example, a CNI plugin), you must add a matching toleration. Without it, the DaemonSet simply skips those nodes.

Update Strategies

DaemonSets support two update strategies that control how pods are replaced when you change the pod template:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1   # update one node at a time
      maxSurge: 0          # don't create new pod before old is terminated

Jobs — Run-to-Completion Workloads

A Job creates one or more pods and ensures that a specified number of them successfully terminate. Once the required number of completions is reached, the Job is considered complete. Unlike Deployments, a Job never restarts a pod that exited with status 0 — the work is done.

Jobs are the right tool for batch processing, data migrations, one-off scripts, report generation, and any task that has a clear beginning and end. The controller handles retries on failure, parallelism, deadlines, and cleanup — you declare the desired behavior, and Kubernetes orchestrates it.

Basic Job YAML

This Job runs a database migration. It creates a single pod, allows up to 4 retries on failure, and enforces a 10-minute deadline.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  backoffLimit: 4
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: myapp/migrations:v2.5.0
          command: ["python", "manage.py", "migrate", "--no-input"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: url

Key Job Configuration Fields

Jobs expose several fields that control execution, failure handling, and lifecycle. Understanding each one prevents common pitfalls like runaway retry loops or orphaned completed pods consuming resources.

Parallel Jobs with Completions

When you need to process multiple independent items — say, rendering 50 video chunks — set completions to the total number of items and parallelism to how many can run concurrently. The Job controller creates new pods as existing ones succeed, until all completions are reached.

apiVersion: batch/v1
kind: Job
metadata:
  name: video-render
spec:
  completions: 50
  parallelism: 10
  backoffLimit: 5
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: renderer
          image: myapp/renderer:v1.2
          command: ["./render.sh"]
          env:
            - name: QUEUE_URL
              value: "sqs://render-jobs"

In this pattern, each pod pulls work from an external queue (SQS, Redis, RabbitMQ). The Job doesn’t know what each pod processes — it only ensures 50 total pods succeed and keeps up to 10 running at any time.

Indexed Jobs for Unique Work Assignment

Indexed Jobs (stable since Kubernetes 1.24) assign each pod a unique index from 0 to completions - 1, exposed via the JOB_COMPLETION_INDEX environment variable. This eliminates the need for an external work queue when each unit of work can be identified by a simple integer — processing file shards, database partition ranges, or test suite splits.

apiVersion: batch/v1
kind: Job
metadata:
  name: data-process
spec:
  completionMode: Indexed
  completions: 8
  parallelism: 4
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: processor
          image: myapp/data-pipeline:v3.0
          command:
            - python
            - process_shard.py
            - --shard-index=$(JOB_COMPLETION_INDEX)
            - --total-shards=8

Pod 0 gets JOB_COMPLETION_INDEX=0, pod 1 gets 1, and so on. If pod 3 fails and is retried, the replacement pod still receives index 3. This guarantees exactly-once processing per index when your application is idempotent.

CronJobs — Scheduled Jobs

A CronJob creates a Job on a repeating schedule, using the same cron expression format that Linux administrators have used for decades. It’s the Kubernetes-native replacement for crontab entries, with the added benefit of running inside the cluster where it has access to Kubernetes secrets, volumes, and service networking.

Schedule Syntax

The schedule field uses standard five-field cron syntax. Each field can be a value, range, step, or wildcard:

Some common schedules for reference: "0 * * * *" (every hour on the hour), "*/15 * * * *" (every 15 minutes), "0 2 * * *" (daily at 2 AM), "0 0 * * 0" (weekly on Sunday at midnight), and "0 0 1 * *" (first day of every month).

Basic CronJob YAML

This CronJob creates a nightly database backup at 2:30 AM. The jobTemplate section is identical to a standalone Job spec — the CronJob controller simply stamps out a new Job on each trigger.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-backup
spec:
  schedule: "30 2 * * *"
  timeZone: "America/New_York"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      activeDeadlineSeconds: 1800
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: myapp/db-backup:v1.4
              command:
                - /bin/sh
                - -c
                - pg_dump $DATABASE_URL | gzip > /backups/db-$(date +%Y%m%d).sql.gz
              env:
                - name: DATABASE_URL
                  valueFrom:
                    secretKeyRef:
                      name: db-credentials
                      key: url
              volumeMounts:
                - name: backup-storage
                  mountPath: /backups
          volumes:
            - name: backup-storage
              persistentVolumeClaim:
                claimName: backup-pvc

CronJob Configuration Fields

Beyond the schedule, CronJobs offer several fields that control overlap behavior, failure tolerance, and history management. Getting these right is important — a misconfigured CronJob can silently skip runs, stack up parallel executions, or leave hundreds of completed Job objects in the cluster.

Choosing a Concurrency Policy

The concurrencyPolicy field is the most consequential CronJob setting. The right choice depends entirely on whether your task is safe to run in parallel and what should happen when a run takes longer than the interval between schedules.

• Allow — Use when each run is independent and overlapping is harmless. Example: sending a periodic health check report where two overlapping reports are fine.
• Forbid — Use when concurrent runs would conflict or corrupt shared resources. Example: a database backup that acquires an advisory lock — running two in parallel would cause one to fail. Missed schedules are simply skipped.
• Replace — Use when only the latest run matters and stale runs should be stopped. Example: a cache warm-up job where the newest data supersedes anything the previous run was building.

Choosing the Right Controller

When you’re modeling a new workload, ask two questions: Does it need to run continuously or to completion? Does it need to run on specific nodes, or wherever the scheduler decides? The answer maps directly to one of the five workload controllers:

Multi-Container Pod Patterns — Init, Sidecar, Ambassador, Adapter

A Pod is not limited to a single container. When you place multiple containers in the same Pod, they share two critical resources: a network namespace (they all reach each other on localhost) and optionally storage volumes (they can read and write the same files). This is the foundation for every multi-container pattern in Kubernetes.

You would never cram unrelated services into one Pod — that defeats the purpose of microservices. Multi-container Pods exist for tightly coupled helpers that genuinely need to share fate and resources with the main application container. Kubernetes formalizes this with four well-established patterns: Init, Sidecar, Ambassador, and Adapter.

flowchart TB
    subgraph pod ["Pod Boundary — shared localhost + volumes"]
        direction TB

        subgraph init ["① Init Containers (sequential, run-to-completion)"]
            I1["init-1: wait-for-db"] --> I2["init-2: run-migrations"]
        end

        subgraph runtime ["② App + Helper Containers (run concurrently)"]
            direction LR
            APP["app container\n:8080"]

            subgraph sidecar ["Sidecar"]
                S["log-shipper\nreads shared volume"]
            end
            subgraph ambassador ["Ambassador"]
                A["db-proxy\nlocalhost:5432"]
            end
            subgraph adapter ["Adapter"]
                D["metrics-adapter\nPrometheus format"]
            end
        end

        init --> runtime
    end

    style pod fill:#1a1a2e,stroke:#6366f1,stroke-width:2px,color:#e2e8f0
    style init fill:#1e293b,stroke:#f59e0b,stroke-width:1px,color:#e2e8f0
    style runtime fill:#1e293b,stroke:#22d3ee,stroke-width:1px,color:#e2e8f0
    style sidecar fill:#0f172a,stroke:#a78bfa,stroke-width:1px,color:#e2e8f0
    style ambassador fill:#0f172a,stroke:#34d399,stroke-width:1px,color:#e2e8f0
    style adapter fill:#0f172a,stroke:#fb923c,stroke-width:1px,color:#e2e8f0
    

Why Containers in the Same Pod?

Containers in the same Pod share a network namespace — they communicate over localhost without any Service or DNS lookup. If your app container listens on port 8080, a sidecar container can reach it at localhost:8080. They also share the same IP address, so external callers see one network identity.

Shared volumes let containers exchange files without network overhead. An app can write log files to an emptyDir volume, and a sidecar can tail those same files and ship them to a logging backend. This cooperation model is what makes the four patterns below so effective.

Init Containers

Init containers run before any app container starts. They execute sequentially — the second init container will not start until the first exits with code 0. If any init container fails, the kubelet retries it according to the Pod's restartPolicy. The Pod stays in Pending state until all init containers succeed.

This makes them perfect for setup tasks that must complete before your application is ready: waiting for a database to accept connections, running schema migrations, cloning a config repository, or downloading ML model weights. The app container is guaranteed that these preconditions are met by the time it starts.

Real-World Example: Wait for a Database, Then Migrate

apiVersion: v1
kind: Pod
metadata:
  name: web-app
spec:
  initContainers:
    # Init container 1: block until PostgreSQL accepts connections
    - name: wait-for-db
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          until nc -z postgres-svc 5432; do
            echo "Waiting for database..."
            sleep 2
          done
          echo "Database is ready"

    # Init container 2: run schema migrations
    - name: run-migrations
      image: myapp/migrator:1.4.0
      command: ["./migrate", "--source", "file:///migrations", "--database", "$(DATABASE_URL)", "up"]
      env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: url

  containers:
    - name: web
      image: myapp/web:2.1.0
      ports:
        - containerPort: 8080
      env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: url

The execution order is strict: wait-for-db loops until PostgreSQL is reachable, then run-migrations applies any pending schema changes, and only then does the web container start. If the migration fails (non-zero exit), the kubelet restarts the init container — the app never starts in a broken state.

Sidecar Containers

Sidecar containers run alongside the main application container for the entire lifetime of the Pod. They extend the app's capabilities without modifying its code — shipping logs, injecting TLS, reloading configuration files, or proxying traffic through a service mesh like Istio's Envoy.

Historically, sidecars were just regular containers listed in spec.containers. The problem: Kubernetes had no way to distinguish a helper from the primary workload. Sidecars could start after the app, and worse, they could prevent a Job Pod from completing because they never exited.

Native Sidecar Support (Kubernetes 1.28+)

Kubernetes 1.28 introduced native sidecar containers (beta in 1.29, stable in 1.31). The mechanism is elegant: you declare a container in initContainers with restartPolicy: Always. This tells the kubelet to start it before the app containers and keep it running for the Pod's entire lifetime. Native sidecars start in init container order, are guaranteed running before app containers launch, and are terminated after the main containers exit.

Real-World Example: Log Shipper Sidecar

apiVersion: v1
kind: Pod
metadata:
  name: app-with-log-shipper
spec:
  initContainers:
    # Native sidecar: starts before app, stays running, stops after app
    - name: log-shipper
      image: fluent/fluent-bit:3.1
      restartPolicy: Always          # This makes it a native sidecar
      volumeMounts:
        - name: app-logs
          mountPath: /var/log/app
      env:
        - name: FLUENTBIT_OUTPUT
          value: "elasticsearch"
        - name: ES_HOST
          value: "elasticsearch.logging.svc.cluster.local"

  containers:
    - name: app
      image: myapp/api:3.0.0
      ports:
        - containerPort: 8080
      volumeMounts:
        - name: app-logs
          mountPath: /var/log/app

  volumes:
    - name: app-logs
      emptyDir: {}

The app container writes structured JSON logs to /var/log/app/. The log-shipper sidecar tails those files and forwards them to Elasticsearch. Because it is declared as a native sidecar (restartPolicy: Always in initContainers), the kubelet guarantees it starts before app and terminates after app exits. No lost log lines on shutdown.

Ambassador Pattern

The Ambassador pattern places a proxy container in the Pod that handles outbound connections on behalf of the app. The app connects to localhost on a well-known port, and the ambassador container handles the complexity of routing, connection pooling, authentication, or protocol translation to the actual remote service.

This cleanly separates connection logic from business logic. The app does not need to know about TLS certificates, connection pool sizes, retry policies, or even which database host it is talking to — it just connects to localhost:5432 and the ambassador handles the rest.

Real-World Example: Cloud SQL Auth Proxy

apiVersion: v1
kind: Pod
metadata:
  name: app-with-db-proxy
spec:
  serviceAccountName: cloud-sql-sa
  containers:
    - name: app
      image: myapp/api:3.0.0
      ports:
        - containerPort: 8080
      env:
        # App connects to localhost — no knowledge of Cloud SQL
        - name: DATABASE_HOST
          value: "127.0.0.1"
        - name: DATABASE_PORT
          value: "5432"
        - name: DATABASE_NAME
          value: "myapp_production"

    # Ambassador: proxies localhost:5432 to Cloud SQL over IAM auth
    - name: cloud-sql-proxy
      image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.13.0
      args:
        - "--structured-logs"
        - "--auto-iam-authn"
        - "--port=5432"
        - "my-project:us-central1:prod-db"
      securityContext:
        runAsNonRoot: true
      resources:
        requests:
          cpu: 50m
          memory: 64Mi

Google's Cloud SQL Auth Proxy is the canonical ambassador example. The app container connects to 127.0.0.1:5432 as if it were a local PostgreSQL server. The cloud-sql-proxy ambassador terminates that connection and establishes an authenticated, encrypted tunnel to the actual Cloud SQL instance. The app needs zero Cloud SQL awareness — no special drivers, no IAM token management, no TLS certificate handling.

Adapter Pattern

The Adapter pattern normalizes or transforms output from the main container into a format that external systems expect. The most common use case is metrics: your application might expose metrics in a proprietary format, and an adapter container converts them into a Prometheus-compatible /metrics endpoint so your monitoring stack can scrape them uniformly.

Like the Ambassador, the Adapter runs alongside the app and communicates over localhost or shared volumes. The difference is directional: Ambassadors proxy outbound traffic, while Adapters transform output data.

Real-World Example: Redis Metrics Adapter for Prometheus

apiVersion: v1
kind: Pod
metadata:
  name: redis-with-exporter
  labels:
    app: redis
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9121"
spec:
  containers:
    - name: redis
      image: redis:7.2-alpine
      ports:
        - containerPort: 6379

    # Adapter: reads Redis INFO command, exposes Prometheus metrics
    - name: redis-exporter
      image: oliver006/redis_exporter:v1.62.0
      ports:
        - containerPort: 9121
      env:
        - name: REDIS_ADDR
          value: "localhost:6379"
      resources:
        requests:
          cpu: 25m
          memory: 32Mi

Redis does not natively expose Prometheus metrics. The redis-exporter adapter container connects to Redis on localhost:6379, runs the INFO command, and translates the output into Prometheus exposition format on port 9121. Prometheus scrapes the adapter, not Redis directly. The same pattern works for MySQL, PostgreSQL, NGINX, and dozens of other services that have community-maintained exporters.

Choosing the Right Pattern

The Kubernetes Networking Model and CNI Plugins

Networking is arguably the most complex piece of a Kubernetes cluster, yet it rests on a deceptively simple foundation. Before a single packet flows, Kubernetes mandates three non-negotiable rules about how Pods, nodes, and the network interact. Understanding these rules — and the pluggable system that implements them — is essential for debugging connectivity issues, choosing the right network plugin, and reasoning about performance.

The Three Fundamental Networking Rules

The Kubernetes networking model is defined by three invariants. These are not suggestions — every conformant cluster implementation must satisfy all three. They are spelled out in the official Kubernetes documentation and form the contract that every CNI plugin must uphold.

1. Every Pod gets its own IP address. Each Pod is assigned a unique, cluster-routable IP from the Pod CIDR range. Containers within the same Pod share this IP and communicate over localhost.
2. Pods can communicate with all other Pods without NAT. Any Pod can reach any other Pod in the cluster using the destination Pod's IP directly. Source and destination addresses are never translated in transit.
3. Nodes can communicate with all Pods (and vice versa) without NAT. A process running on a node can reach any Pod IP directly, and Pods can reach node IPs without address translation.

The result is a flat network — every Pod and node exists in a single, shared address space. There is no port-mapping layer between Pods, no manual link configuration, and no NAT rewriting packet headers. If Pod A knows Pod B's IP, it can just send a packet there.

How This Differs from Docker's Default Networking

If you are coming from Docker, this model will feel unfamiliar. Docker's default bridge networking takes a fundamentally different approach: containers on a single host share a private bridge network (typically 172.17.0.0/16), and communication with the outside world requires explicit port mapping via -p flags. This means two containers on different hosts cannot talk to each other by default — they need port forwarding, link aliases, or an overlay network like Docker Swarm's.

Kubernetes chose the flat model because it eliminates an entire class of problems. Port conflicts disappear. Applications do not need to know whether they are running in a container. Network policies can reason about Pod IPs as stable identities rather than chasing ephemeral port mappings.

How Pods Get Their IPs — Under the Hood

When the kubelet on a node needs to start a new Pod, the sequence works like this: the container runtime creates the Pod's network namespace (an isolated Linux network stack), then calls the configured CNI plugin. The plugin assigns an IP address from the node's allocated Pod CIDR subnet, creates a virtual ethernet (veth) pair, connects one end to the Pod's namespace and the other to the host network, and sets up routes so traffic can flow.

You can verify a Pod's IP and network namespace from the node itself:

# Get the IP assigned to a Pod
kubectl get pod my-app -o wide
# NAME     READY   STATUS    IP            NODE
# my-app   1/1     Running   10.244.1.23   worker-01

# From worker-01, inspect the veth pair
ip link show type veth
# You'll see interfaces like cali* (Calico), lxc* (Cilium), or vethXXXX

# Trace the route to a Pod on another node
ip route get 10.244.2.45
# 10.244.2.45 via 192.168.1.12 dev eth0  (routed to worker-02)

Pod-to-Pod Communication Across Nodes

When Pod A on Node 1 sends a packet to Pod B on Node 2, the packet travels through several layers. It exits Pod A's network namespace via the veth pair, hits the host network stack on Node 1, gets routed to Node 2 (via overlay encapsulation or direct routing depending on the CNI plugin), enters Node 2's host network stack, and is finally delivered into Pod B's namespace.

flowchart LR
    subgraph Node1["Node 1 (192.168.1.11)"]
        direction TB
        PodA["Pod A\n10.244.1.23"]
        veth1["veth pair"]
        Host1["Host Network Stack\n+ routing table"]
        PodA --- veth1 --- Host1
    end

    subgraph Node2["Node 2 (192.168.1.12)"]
        direction TB
        Host2["Host Network Stack\n+ routing table"]
        veth2["veth pair"]
        PodB["Pod B\n10.244.2.45"]
        Host2 --- veth2 --- PodB
    end

    Host1 -- "Overlay (VXLAN/GENEVE)\nor BGP direct route" --> Host2

    style PodA fill:#4a9eff,stroke:#2d7cd4,color:#fff
    style PodB fill:#4a9eff,stroke:#2d7cd4,color:#fff
    style Host1 fill:#f0f4f8,stroke:#9aa5b4
    style Host2 fill:#f0f4f8,stroke:#9aa5b4
    style veth1 fill:#ffd43b,stroke:#e6b800,color:#333
    style veth2 fill:#ffd43b,stroke:#e6b800,color:#333
        

The critical detail is that Pod A's source IP is preserved — Pod B sees the real IP of Pod A, not a translated address. This is what makes network policies, access logs, and distributed tracing work correctly.

CNI — The Container Network Interface

CNI is a specification, not a piece of software. It defines a minimal contract between a container runtime and a network plugin: "here's a network namespace, set it up" and "here's a network namespace, tear it down." The spec was originally developed by CoreOS and is now maintained by the CNCF. It is used by Kubernetes, but also by other container orchestrators like Podman and CRI-O directly.

A CNI plugin is just a binary that the container runtime executes. The runtime passes information through environment variables and stdin (a JSON configuration), and the plugin responds by configuring the network namespace and returning the result (including the assigned IP) as JSON on stdout.

Plugin Configuration

CNI plugins are configured via two things on each node:

• Plugin binaries in /opt/cni/bin/ — the actual executables (e.g., calico, flannel, bridge, loopback).
• Configuration files in /etc/cni/net.d/ — JSON or conflist files that tell the runtime which plugin to invoke and with what parameters.

The container runtime (containerd, CRI-O) reads the configuration directory, picks the first file alphabetically, and uses it for all Pod network setup. Here is a typical configuration file:

{
  "cniVersion": "1.0.0",
  "name": "k8s-pod-network",
  "type": "calico",
  "ipam": {
    "type": "calico-ipam"
  },
  "policy": {
    "type": "k8s"
  },
  "log_level": "info"
}

The type field maps directly to a binary name in /opt/cni/bin/. The ipam block configures IP address management — how the plugin allocates and tracks Pod IPs. Many plugins include their own IPAM, but you can also use standalone IPAM plugins like host-local or whereabouts.

The Plugin Lifecycle: ADD, DEL, CHECK

The CNI spec defines three operations that a runtime can invoke on a plugin. These map directly to the lifecycle of a Pod's network namespace:

You can inspect what happens during these operations by looking at kubelet logs on a node when a Pod is scheduled:

# Watch CNI activity in kubelet logs
journalctl -u kubelet -f | grep -i cni

# List installed CNI plugins on a node
ls /opt/cni/bin/
# bandwidth  bridge  calico  calico-ipam  flannel  host-local  loopback  portmap

# View active CNI configuration
cat /etc/cni/net.d/10-calico.conflist | jq '.plugins[].type'
# "calico"
# "bandwidth"
# "portmap"

Comparing Popular CNI Plugins

The CNI plugin you choose has a direct impact on performance, features, operational complexity, and what network policies you can enforce. There is no single "best" plugin — the right choice depends on your cluster size, performance requirements, and whether you need advanced features like encryption or deep observability.

Calico

Calico is the most widely deployed CNI plugin in production Kubernetes clusters. In its default mode, it uses BGP (Border Gateway Protocol) to distribute Pod routes directly across nodes — no encapsulation overhead, no tunnel interfaces. Each node acts as a BGP peer and announces its Pod CIDR to the rest of the cluster. This approach gives near bare-metal networking performance.

When BGP is not feasible (for example, in cloud VPCs that block BGP), Calico falls back to VXLAN overlay mode. Calico also includes the most mature network policy implementation in the ecosystem, supporting both standard Kubernetes NetworkPolicy resources and its own extended GlobalNetworkPolicy CRD for cluster-wide rules.

# Install Calico (operator-based)
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/custom-resources.yaml

# Verify Calico Pods are running
kubectl get pods -n calico-system
# NAME                                       READY   STATUS    RESTARTS
# calico-kube-controllers-7c5f8db89c-x2g4l   1/1     Running   0
# calico-node-abcde                           1/1     Running   0
# calico-typha-6f8b5c9d4f-k8m2n              1/1     Running   0

# Check BGP peering status
sudo calicoctl node status
# IPv4 BGP status
# +--------------+-------------------+-------+----------+-------------+
# | PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
# +--------------+-------------------+-------+----------+-------------+
# | 192.168.1.12 | node-to-node mesh | up    | 10:23:45 | Established |
# +--------------+-------------------+-------+----------+-------------+

Cilium

Cilium takes a radically different approach by moving networking logic into the Linux kernel using eBPF (extended Berkeley Packet Filter). Instead of configuring iptables rules (which become a bottleneck at scale), Cilium attaches eBPF programs directly to network interfaces. This results in lower latency, higher throughput, and — crucially — L7 visibility into application-layer protocols.

Cilium can enforce network policies that understand HTTP methods, gRPC services, Kafka topics, and DNS queries — not just IP addresses and ports. It also includes Hubble, an observability platform that gives you a real-time network traffic flow map of your entire cluster.

# Install Cilium via Helm
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.15.0 \
  --namespace kube-system \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

# Check Cilium status
cilium status
# Cilium health daemon:  Ok
# IPAM:                  IPv4: 12/254 allocated
# BandwidthManager:      Disabled
# Encryption:            Disabled

# Observe live traffic flows with Hubble
hubble observe --namespace default --protocol HTTP
# TIMESTAMP             SOURCE             DESTINATION          TYPE     VERDICT
# Jan 15 10:00:01.234   default/frontend   default/api-server   L7/HTTP  FORWARDED
#                        GET /api/v1/users => 200

Flannel

Flannel is the simplest CNI plugin and often the first one people encounter. It creates a VXLAN overlay network by default: each node gets a subnet from the cluster's Pod CIDR, and traffic between nodes is encapsulated in VXLAN packets. There is no support for network policies — if you need policies with Flannel, you typically pair it with Calico in a configuration called "Canal."

Flannel's simplicity is both its strength and its limitation. It is ideal for development clusters, CI/CD environments, and learning Kubernetes. For production workloads where you need policy enforcement or performance optimization, you will outgrow it.

# Install Flannel
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# Verify — Flannel runs as a DaemonSet on every node
kubectl get pods -n kube-flannel
# NAME                    READY   STATUS    RESTARTS
# kube-flannel-ds-abc12   1/1     Running   0
# kube-flannel-ds-def34   1/1     Running   0

# Inspect the VXLAN interface Flannel creates
ip -d link show flannel.1
# flannel.1: <BROADCAST,MULTICAST,UP> mtu 1450 ...
#     vxlan id 1 ... dstport 8472

Weave Net

Weave Net builds a mesh overlay network between nodes using VXLAN (fast datapath) or a user-space "sleeve" mode that can traverse firewalls and NATs. Its standout feature is zero-configuration encryption — enable it with a single password and all inter-node traffic is encrypted via IPsec. Weave also supports basic Kubernetes network policies.

Weave is a solid choice for small- to medium-sized clusters where ease of setup and built-in encryption matter more than raw performance. However, it has seen less active development compared to Calico and Cilium in recent years.

Debugging CNI Issues

When a Pod is stuck in ContainerCreating and the events show a CNI error, the problem is almost always in one of three places: the CNI binary is missing, the configuration file is malformed, or the IPAM pool is exhausted. Here is a quick diagnostic checklist:

# 1. Check if the CNI config exists
ls /etc/cni/net.d/
# If empty, no CNI plugin is installed — Pods will stay in ContainerCreating

# 2. Check if the CNI binary exists
ls /opt/cni/bin/ | grep calico
# If the type in your config doesn't match a binary here, ADD will fail

# 3. Look at kubelet logs for the specific error
journalctl -u kubelet --since "5 min ago" | grep -i "cni\|network"

# 4. Check Pod events for the error message
kubectl describe pod stuck-pod | tail -20
# Warning  FailedCreatePodSandBox  kubelet  ...
# failed to set up sandbox container network:
# plugin type="calico" failed: ...

# 5. Verify the CNI plugin DaemonSet is healthy
kubectl get ds -n calico-system   # or kube-flannel, kube-system, etc.

Services — ClusterIP, NodePort, and LoadBalancer

Every Pod in Kubernetes gets its own IP address, but that IP is ephemeral. When a Pod is killed and replaced — whether by a Deployment rollout, a node failure, or a scaling event — the new Pod receives a completely different IP. Any client that was using the old IP now has a broken connection. This is the fundamental problem that Services solve.

A Service provides a stable virtual IP address (called a ClusterIP) and a DNS name that remain constant for the lifetime of the Service object. Behind the scenes, the Service uses label selectors to track which Pods should receive traffic, and kube-proxy programs networking rules on every node to route packets to healthy backends. The result is a reliable, load-balanced endpoint that decouples clients from the volatile lifecycle of Pods.

ClusterIP — The Default Service Type

When you create a Service without specifying a type, you get a ClusterIP service. Kubernetes allocates a virtual IP from the cluster’s service CIDR range (configured at cluster creation, e.g., 10.96.0.0/12). This IP is not bound to any network interface or node — it exists only in the cluster’s networking rules. Pods and other Services within the cluster can reach it, but nothing outside the cluster can.

The virtual IP acts as a stable front door. When a packet is sent to the ClusterIP on a given port, kube-proxy intercepts it and forwards it to one of the backing Pods. The selection of which Pod receives the packet depends on the proxy mode in use.

apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: ecommerce
spec:
  type: ClusterIP          # default — can be omitted
  selector:
    app: order-api
    tier: backend
  ports:
    - name: http
      protocol: TCP
      port: 80              # port exposed on the ClusterIP
      targetPort: 8080      # port the Pods are listening on

The selector field is what links the Service to its Pods. Kubernetes continuously watches for Pods matching the labels app: order-api and tier: backend, and adds their IPs to the Service’s endpoint list. When a Pod becomes unready or is deleted, it is removed automatically.

How kube-proxy Makes It Work

kube-proxy runs on every node and watches the API server for Service and Endpoint changes. When it detects an update, it programs the node’s networking layer to perform the actual packet rewriting. There are three proxy modes, with iptables being the most common.

In iptables mode, kube-proxy creates a chain of rules for each Service. A packet destined for the ClusterIP is DNAT’d (destination NAT) to a randomly selected Pod IP. Return traffic is automatically reverse-NAT’d via conntrack, so the client sees responses from the ClusterIP — not the Pod IP. In IPVS mode, the same concept applies but the kernel’s built-in load balancer handles the forwarding with better performance at scale.

NodePort — Exposing Services Outside the Cluster

A NodePort service builds on top of ClusterIP. Kubernetes still allocates a virtual ClusterIP, but additionally opens a static port — the NodePort — on every node in the cluster. Any traffic arriving at <NodeIP>:<NodePort> is forwarded to the Service, which then load-balances it to a backing Pod. The NodePort range is 30000–32767 by default (configurable via the API server’s --service-node-port-range flag).

This means external clients can reach the service by hitting any node’s IP on the allocated port. The node does not need to be running the backing Pod — kube-proxy on every node has rules to forward NodePort traffic to the correct Pod, even if it is on a different node.

apiVersion: v1
kind: Service
metadata:
  name: order-service-nodeport
spec:
  type: NodePort
  selector:
    app: order-api
  ports:
    - name: http
      protocol: TCP
      port: 80              # ClusterIP port (internal)
      targetPort: 8080      # Pod port
      nodePort: 30080       # static port on every node (optional — auto-assigned if omitted)

With this Service, traffic flows through three layers: the external client connects to 192.168.1.10:30080 (any node IP), kube-proxy forwards it to the ClusterIP 10.96.x.x:80, and then DNAT sends it to a Pod on port 8080. If you omit the nodePort field, Kubernetes auto-assigns one from the available range.

LoadBalancer — Cloud-Integrated External Access

The LoadBalancer type extends NodePort by instructing the cloud provider’s controller to provision an external load balancer (an AWS NLB/ALB, GCP TCP LB, Azure LB, etc.). The cloud LB receives a public or internal IP and forwards traffic to the NodePorts on your cluster nodes. Kubernetes automatically configures health checks and backend pools.

This is the simplest way to expose a Service to the internet in a cloud environment. However, each LoadBalancer Service gets its own cloud LB — which means its own IP address and its own billing line item. For clusters with many externally-facing services, an Ingress controller (covered in the next section) is more cost-effective.

apiVersion: v1
kind: Service
metadata:
  name: order-service-lb
  annotations:
    # AWS-specific: request an NLB instead of a Classic LB
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  selector:
    app: order-api
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
  # Optional: restrict source IPs allowed through the LB
  loadBalancerSourceRanges:
    - 203.0.113.0/24

After creation, the status.loadBalancer.ingress field is populated with the external IP or hostname assigned by the cloud provider. This can take 30 seconds to a few minutes depending on the cloud. Use kubectl get svc order-service-lb -w to watch for it.

ExternalName — DNS Alias for External Services

ExternalName is fundamentally different from the other three types. It does not create a ClusterIP, does not configure kube-proxy rules, and does not proxy any traffic. Instead, it creates a DNS CNAME record in the cluster’s DNS (CoreDNS) that maps the Service name to an external hostname.

apiVersion: v1
kind: Service
metadata:
  name: legacy-payments
  namespace: ecommerce
spec:
  type: ExternalName
  externalName: payments.legacy-datacenter.example.com

When a Pod resolves legacy-payments.ecommerce.svc.cluster.local, CoreDNS returns a CNAME to payments.legacy-datacenter.example.com. This is useful during migrations — your application code references a Kubernetes Service name, and you can later swap the ExternalName for a ClusterIP Service pointing to in-cluster Pods without changing application configuration.

Traffic Flow Through Each Service Type

The following diagram shows how traffic traverses the networking layers for each Service type. Notice how each type builds on the previous one — LoadBalancer wraps NodePort, which wraps ClusterIP.

flowchart LR
    subgraph External
        Client(["External Client"])
        CloudLB(["Cloud Load Balancer"])
    end

    subgraph Cluster ["Kubernetes Cluster"]
        subgraph Node1 ["Node 1 — 192.168.1.10"]
            KP1["kube-proxy
iptables / IPVS rules"]
            Pod1(["Pod A
10.244.1.5:8080"])
        end
        subgraph Node2 ["Node 2 — 192.168.1.11"]
            KP2["kube-proxy
iptables / IPVS rules"]
            Pod2(["Pod B
10.244.2.8:8080"])
        end
        SvcIP["ClusterIP
10.96.47.12:80"]
    end

    Client -- "1 LoadBalancer
external-ip:80" --> CloudLB
    CloudLB -- "2 NodePort
any-node:30080" --> KP1
    KP1 -- "3 ClusterIP
10.96.47.12:80" --> SvcIP
    SvcIP -. "DNAT to Pod" .-> Pod1
    SvcIP -. "DNAT to Pod" .-> Pod2
    KP2 -- "3 ClusterIP" --> SvcIP

    InternalPod(["In-Cluster Pod"]) -- "ClusterIP only
10.96.47.12:80" --> SvcIP

    style SvcIP fill:#4a90d9,color:#fff,stroke:#2a6cb8
    style CloudLB fill:#f5a623,color:#fff,stroke:#d4891a
    style Pod1 fill:#7ed321,color:#fff,stroke:#5ea318
    style Pod2 fill:#7ed321,color:#fff,stroke:#5ea318
    

Endpoints and EndpointSlices

When you create a Service with a selector, Kubernetes automatically creates a companion Endpoints object with the same name. This object contains a flat list of IP:port pairs for every Pod that matches the selector and has passed its readiness probe. kube-proxy watches Endpoints objects to know where to send traffic.

# View the Endpoints for a Service
kubectl get endpoints order-service -n ecommerce
# NAME            ENDPOINTS                                AGE
# order-service   10.244.1.5:8080,10.244.2.8:8080          4m

# Detailed view shows ready and not-ready addresses
kubectl describe endpoints order-service -n ecommerce

The problem with Endpoints is scalability. A single Endpoints object stores every backend IP in one resource. For Services with thousands of Pods, any single Pod change triggers an update to the entire Endpoints object, which must be transmitted to every node running kube-proxy. This creates a quadratic scaling problem.

EndpointSlices (stable since Kubernetes v1.21) fix this by splitting the backend list into multiple slices, each holding up to 100 endpoints by default. When a Pod changes, only the affected EndpointSlice is updated and propagated. This dramatically reduces API server load and network bandwidth in large clusters.

# List EndpointSlices for a Service
kubectl get endpointslices -l kubernetes.io/service-name=order-service -n ecommerce

# Inspect a specific EndpointSlice
kubectl describe endpointslice order-service-abc12 -n ecommerce

Headless Services

Sometimes you do not want Kubernetes to load-balance for you. You want the actual Pod IPs — for example, when running a database cluster where each node has a distinct identity, or when implementing client-side load balancing with gRPC. A headless Service is created by setting clusterIP: None.

With a headless Service, Kubernetes does not allocate a virtual IP and kube-proxy does not create any forwarding rules. Instead, a DNS lookup for the Service name returns A records for each individual Pod IP. If the Service is combined with a StatefulSet, each Pod also gets a stable DNS hostname like pod-0.my-service.namespace.svc.cluster.local.

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  type: ClusterIP
  clusterIP: None            # makes it headless
  selector:
    app: postgres
  ports:
    - name: tcp-postgres
      port: 5432
      targetPort: 5432

# DNS lookup returns individual Pod IPs (no ClusterIP)
kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never --   nslookup postgres-headless.default.svc.cluster.local
# Server:    10.96.0.10
# Name:      postgres-headless.default.svc.cluster.local
# Address 1: 10.244.1.12
# Address 2: 10.244.2.19
# Address 3: 10.244.3.7

Session Affinity

By default, kube-proxy distributes traffic to backends with no stickiness — each new connection may land on a different Pod. If your application requires that all requests from the same client go to the same Pod (e.g., for in-memory session state), you can enable session affinity.

Kubernetes supports one type of session affinity: ClientIP. When enabled, kube-proxy creates affinity rules based on the client’s source IP address. All connections from the same IP are routed to the same Pod for a configurable timeout (default: 10,800 seconds / 3 hours).

apiVersion: v1
kind: Service
metadata:
  name: session-app
spec:
  selector:
    app: web-frontend
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600    # 1-hour sticky sessions
  ports:
    - port: 80
      targetPort: 3000

Internal and External Traffic Policies

By default, when traffic arrives at a node, kube-proxy may forward it to a Pod on any node in the cluster. This adds an extra network hop and obscures the client’s source IP (because the packet is SNAT’d by the forwarding node). Two fields let you control this behavior.

externalTrafficPolicy

For NodePort and LoadBalancer Services, setting externalTrafficPolicy: Local tells kube-proxy to only forward to Pods running on the same node that received the traffic. This preserves the client’s source IP and eliminates the extra hop, but it means nodes without matching Pods will fail health checks and receive no traffic from the load balancer. You must ensure your Pods are reasonably spread across nodes.

internalTrafficPolicy

Similarly, internalTrafficPolicy: Local (available since Kubernetes v1.26) restricts in-cluster traffic to Pods on the same node as the client. This is useful for node-local caches or logging agents where you want each Pod to talk to the agent on its own node.

apiVersion: v1
kind: Service
metadata:
  name: order-service-local
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local    # preserve source IP, avoid extra hops
  internalTrafficPolicy: Cluster  # default — route to any node
  selector:
    app: order-api
  ports:
    - port: 80
      targetPort: 8080

Service Type Comparison

Essential kubectl Commands for Services

# Create a ClusterIP service imperatively
kubectl expose deployment order-api --port=80 --target-port=8080 --name=order-service

# Create a NodePort service
kubectl expose deployment order-api --type=NodePort --port=80 --target-port=8080

# List all Services with wide output (shows ClusterIP + external IP)
kubectl get svc -o wide

# Watch for LoadBalancer external IP assignment
kubectl get svc order-service-lb -w

# Debug: check which Pods are behind a Service
kubectl get endpoints order-service
kubectl get endpointslices -l kubernetes.io/service-name=order-service

# Debug: verify kube-proxy iptables rules for a ClusterIP
sudo iptables -t nat -L KUBE-SERVICES -n | grep order-service

# Temporary local access: port-forward through a Service
kubectl port-forward svc/order-service 8080:80

# DNS resolution test from inside the cluster
kubectl run dns-debug --rm -it --image=busybox:1.36 --restart=Never --   nslookup order-service.ecommerce.svc.cluster.local

Ingress, Ingress Controllers, and the Gateway API

In the previous section, you saw how Services expose Pods inside and outside the cluster via ClusterIP, NodePort, and LoadBalancer. These abstractions work well for raw TCP/UDP connectivity — but they fall apart the moment you need HTTP-aware routing. A LoadBalancer Service gives you a single external IP mapped to a single backend. If you run 20 microservices, you get 20 cloud load balancers, 20 public IPs, and 20 separate bills.

Real-world HTTP traffic demands more: routing requests to different backends based on the hostname (api.example.com vs. app.example.com) or URL path (/api/v1 vs. /static), terminating TLS at the edge, injecting headers, enforcing rate limits, and performing canary releases. These are Layer 7 concerns, and Services operate at Layer 4. This gap is exactly what Ingress — and its successor, the Gateway API — exist to fill.

The Ingress Resource

An Ingress is a Kubernetes API object that declares HTTP and HTTPS routing rules. You define which hostnames and paths map to which backend Services, and an Ingress controller (a separate component you must install) reads those rules and configures a reverse proxy accordingly. The Ingress resource itself does nothing without a controller — it is purely declarative configuration.

Here is a minimal Ingress that routes traffic for two hostnames to two different Services:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 80
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-service
                port:
                  number: 80

The spec.ingressClassName field tells Kubernetes which Ingress controller should handle this resource. Before Kubernetes 1.18, this was done via the kubernetes.io/ingress.class annotation — you will still see that pattern in older configurations.

Path Types

Kubernetes supports three pathType values, and the distinction matters for how requests are matched:

TLS Termination

Ingress supports TLS termination at the edge by referencing a Kubernetes Secret that contains the certificate and private key. The controller terminates HTTPS and forwards plain HTTP to your backend Services.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-ingress
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls-secret
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 80

The Secret must be of type kubernetes.io/tls and contain tls.crt and tls.key data fields. In practice, most teams use cert-manager to automatically provision and renew TLS certificates from Let's Encrypt, eliminating manual Secret management entirely.

Annotations: The Escape Hatch

The Ingress spec is deliberately minimal — it covers hosts, paths, backends, and TLS. Everything else (rate limiting, CORS headers, authentication, WebSocket support, custom timeouts) must be configured through controller-specific annotations. This is the Ingress API's biggest practical reality: annotations are where most of your configuration lives.

metadata:
  annotations:
    # NGINX Ingress Controller annotations
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/rate-limit-window: "1m"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.example.com"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

Comparing Popular Ingress Controllers

Kubernetes does not ship with an Ingress controller. You must install one, and the choice significantly affects your operational experience. Each controller is a reverse proxy that watches for Ingress resources and translates them into its own configuration format.

Limitations of the Ingress API

After years of production use, the Kubernetes community identified several fundamental problems with the Ingress API that no amount of annotation hacking could fix:

• Lowest common denominator spec. The Ingress spec only covers basic host/path routing and TLS. Every controller extends it differently through annotations, making manifests non-portable.
• No role separation. A single Ingress resource mixes infrastructure concerns (which ports to listen on, what TLS policy to use) with application concerns (which paths route where). Both the platform team and the app developer edit the same object.
• No support for non-HTTP protocols. TCP, UDP, and gRPC routing require controller-specific CRDs or annotations — there is no standard way to express them.
• Header-based routing is impossible. You cannot route based on HTTP headers, query parameters, or request methods in the Ingress spec.
• Traffic splitting is absent. Canary deployments, A/B testing, and weighted routing all require annotations or custom CRDs.
• Single resource, single namespace. An Ingress resource can only reference backend Services in its own namespace, making cross-namespace routing cumbersome.

These limitations led to the development of the Gateway API — not as a patch to Ingress, but as a ground-up redesign.

The Gateway API

The Gateway API is a collection of Kubernetes CRDs that provide expressive, extensible, and role-oriented routing. It graduated to GA (v1.0) in October 2023 for its core HTTP routing features. Unlike Ingress, the Gateway API was designed by a multi-vendor working group with explicit goals: portability across implementations, a rich feature set without annotations, and clear separation of concerns between personas.

The Three-Resource Model

The Gateway API splits what was a single Ingress resource into three distinct resource types, each managed by a different persona:

flowchart TB
    subgraph infra["Infrastructure Provider"]
        GC["GatewayClass
Defines the controller implementation
e.g., Envoy, NGINX, Cilium"]
    end

    subgraph platform["Cluster Operator / Platform Team"]
        GW["Gateway
Declares listeners: ports, protocols, TLS
e.g., HTTPS on port 443"]
    end

    subgraph appdev["Application Developer"]
        HR["HTTPRoute
Defines host/path matching,
backends, traffic splitting"]
    end

    GC -->|"referenced by"| GW
    GW -->|"routes attach to"| HR
    

This separation is powerful. The platform team configures TLS policies and which namespaces are allowed to bind routes. App developers create HTTPRoutes in their own namespace without needing to touch infrastructure configuration or request cluster-admin privileges. Each persona controls only what they own.

Gateway API in Practice

Here is a complete example showing all three resources working together. The GatewayClass is typically provided by the controller installation; you will usually only create the Gateway and HTTPRoute.

# 1. GatewayClass — installed by the infrastructure provider
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

# 2. Gateway — created by the platform team
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main-gateway
  namespace: infra
spec:
  gatewayClassName: eg
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-tls
            kind: Secret
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: "true"
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Same

# 3. HTTPRoute — created by the app developer
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: store-routes
  namespace: store
spec:
  parentRefs:
    - name: main-gateway
      namespace: infra
  hostnames:
    - "store.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      backendRefs:
        - name: store-api
          port: 8080
          weight: 90
        - name: store-api-canary
          port: 8080
          weight: 10
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: store-frontend
          port: 3000

Notice the weight fields on the first rule — 90% of /api traffic goes to the stable backend and 10% to the canary. This traffic splitting is a first-class feature of the Gateway API, not an annotation hack. Also note that the HTTPRoute lives in the store namespace but attaches to a Gateway in the infra namespace via parentRefs. Cross-namespace routing is built in.

HTTPRoute Features Beyond Basic Routing

The HTTPRoute resource supports matching and filtering capabilities that are impossible in the Ingress API without annotations:

rules:
  # Header-based routing
  - matches:
      - headers:
          - name: x-api-version
            value: "v2"
        path:
          type: PathPrefix
          value: /api
    backendRefs:
      - name: api-v2
        port: 8080

  # Request redirect
  - matches:
      - path:
          type: Exact
          value: /old-page
    filters:
      - type: RequestRedirect
        requestRedirect:
          scheme: https
          statusCode: 301
          path:
            type: ReplaceFullPath
            replaceFullPath: /new-page

  # Header modification
  - matches:
      - path:
          type: PathPrefix
          value: /internal
    filters:
      - type: RequestHeaderModifier
        requestHeaderModifier:
          add:
            - name: X-Internal-Request
              value: "true"
          remove:
            - X-Debug
    backendRefs:
      - name: internal-service
        port: 8080

Other Route Types

The Gateway API is not limited to HTTP. Alongside HTTPRoute, the specification defines additional route types for other protocols:

Ingress vs. Gateway API: Side-by-Side

Installing and Using the Gateway API

The Gateway API CRDs are not bundled with Kubernetes itself. You install them separately before creating any Gateway or HTTPRoute resources.

# Install the Gateway API CRDs (standard channel — GA resources only)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

# Or include experimental resources (TCPRoute, UDPRoute, TLSRoute)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/experimental-install.yaml

# Verify the CRDs are installed
kubectl get crds | grep gateway.networking.k8s.io

After installing the CRDs, you still need a controller that implements them. Envoy Gateway, Cilium, Istio, and the NGINX Gateway Fabric are all popular choices. Each controller installs its own GatewayClass — check the controller's documentation for setup instructions.

In the next section, you will see how DNS resolution works inside the cluster via CoreDNS — the mechanism that lets Pods discover Services by name, which is ultimately what Ingress and Gateway backends rely on to forward traffic.

DNS Resolution in Kubernetes with CoreDNS

Every time a Pod connects to a Service by name — curl http://api-server or a database connection string like postgres://db-primary:5432 — it relies on DNS resolution happening inside the cluster. Kubernetes uses CoreDNS as its default cluster DNS server (replacing kube-dns since v1.13). CoreDNS runs as a Deployment in the kube-system namespace and provides name resolution for Services, Pods, and external domains to every workload in the cluster.

Understanding how cluster DNS works is essential because DNS misconfiguration is one of the most common causes of connectivity failures in Kubernetes. A surprising amount of latency problems also trace back to how DNS search domains and the ndots setting interact with external lookups.

How CoreDNS Is Deployed

CoreDNS runs as a Deployment (typically two replicas for high availability) behind a Service named kube-dns in the kube-system namespace. The Service gets a well-known ClusterIP — usually 10.96.0.10 or whatever the first usable IP in your Service CIDR is. This IP is critical because the kubelet on every node writes it into the /etc/resolv.conf of every Pod it creates.

# Inspect the CoreDNS Deployment and Service
kubectl get deploy coredns -n kube-system
kubectl get svc kube-dns -n kube-system

# Check what a Pod sees as its DNS config
kubectl exec -it my-pod -- cat /etc/resolv.conf

A typical Pod's /etc/resolv.conf looks like this:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Three things to note here: the nameserver points to the kube-dns Service ClusterIP, the search line lists domain suffixes that get appended during resolution, and ndots:5 controls when those suffixes are used. All three are configured by the kubelet based on the Pod's namespace and DNS policy.

sequenceDiagram
    participant Pod
    participant resolv as /etc/resolv.conf
    participant CoreDNS as CoreDNS (kube-dns)
    participant Upstream as Upstream DNS

    Pod->>resolv: resolve "api-server"
    resolv->>CoreDNS: api-server.default.svc.cluster.local?
    CoreDNS->>CoreDNS: Look up in cluster zone
    CoreDNS-->>Pod: 10.100.23.45 (ClusterIP)

    Note over Pod,Upstream: External domain resolution

    Pod->>resolv: resolve "api.github.com"
    resolv->>CoreDNS: api.github.com.default.svc.cluster.local?
    CoreDNS-->>resolv: NXDOMAIN
    resolv->>CoreDNS: api.github.com.svc.cluster.local?
    CoreDNS-->>resolv: NXDOMAIN
    resolv->>CoreDNS: api.github.com.cluster.local?
    CoreDNS-->>resolv: NXDOMAIN
    resolv->>CoreDNS: api.github.com?
    CoreDNS->>Upstream: api.github.com?
    Upstream-->>CoreDNS: 140.82.121.6
    CoreDNS-->>Pod: 140.82.121.6
    

DNS Naming Conventions

Kubernetes creates DNS records automatically when you create Services and (optionally) Pods. The naming follows a strict hierarchy rooted at cluster.local (the default cluster domain). Understanding these patterns lets you address any resource by its fully qualified domain name (FQDN).

Service DNS Records

Every Service gets an A/AAAA record in the format:

<service-name>.<namespace>.svc.cluster.local

For a ClusterIP Service, this resolves to the virtual ClusterIP. For a headless Service (clusterIP: None), it resolves to the set of Pod IPs backing the Service. Headless Services also create individual A records for each Pod when used with a StatefulSet:

# Regular Service
my-service.production.svc.cluster.local → 10.100.23.45 (ClusterIP)

# Headless Service with StatefulSet
redis-0.redis-headless.production.svc.cluster.local → 10.244.1.12 (Pod IP)
redis-1.redis-headless.production.svc.cluster.local → 10.244.2.8  (Pod IP)
redis-2.redis-headless.production.svc.cluster.local → 10.244.3.19 (Pod IP)

Services with named ports also get SRV records, which encode both the port number and the protocol:

# SRV record format
_<port-name>._<protocol>.<service>.<namespace>.svc.cluster.local

# Example: Service with port named "http" on TCP
_http._tcp.my-service.production.svc.cluster.local → 0 0 8080 my-service.production.svc.cluster.local

Pod DNS Records

Pods get DNS records based on their IP address, with dots replaced by dashes:

# Pod DNS record format
<pod-ip-dashed>.<namespace>.pod.cluster.local

# Example: Pod with IP 10.244.1.12 in "production" namespace
10-244-1-12.production.pod.cluster.local → 10.244.1.12

In practice, you rarely query Pod DNS records directly. Service DNS is the primary mechanism for discovery. The table below summarizes the key record types.

DNS Policies

Not every Pod should resolve DNS the same way. A Pod running a node-level agent might need the host's DNS, while a regular application Pod should use cluster DNS. Kubernetes provides four DNS policies, set via the dnsPolicy field in the Pod spec, that control how /etc/resolv.conf is constructed.

The None policy is often paired with dnsConfig to build a completely custom resolver setup. Here's an example that uses a corporate DNS server alongside CoreDNS:

apiVersion: v1
kind: Pod
metadata:
  name: custom-dns-pod
spec:
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 10.96.0.10        # CoreDNS
      - 172.16.0.53       # Corporate DNS
    searches:
      - default.svc.cluster.local
      - svc.cluster.local
      - corp.example.com
    options:
      - name: ndots
        value: "3"
  containers:
    - name: app
      image: nginx:1.27

The Corefile — CoreDNS Configuration

CoreDNS is configured through a file called the Corefile, stored in a ConfigMap named coredns in the kube-system namespace. The Corefile is a chain of server blocks, each declaring a DNS zone and the plugins that handle it. Plugins execute in the order they are listed.

# View the current Corefile
kubectl get configmap coredns -n kube-system -o yaml

A typical default Corefile looks like this:

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
        max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

Key Plugins Explained

Customizing the Corefile

Two common customizations are forwarding specific domains to custom DNS servers and adding extra DNS entries. You edit the coredns ConfigMap directly. The reload plugin picks up changes automatically.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health { lameduck 5s }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4 {
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
    # Forward corp.example.com to internal DNS
    corp.example.com:53 {
        errors
        cache 30
        forward . 172.16.0.53 172.16.0.54
    }

The second server block above handles all queries for corp.example.com and its subdomains, forwarding them to corporate DNS servers at 172.16.0.53 and 172.16.0.54. Everything else goes through the main block. This pattern is common in hybrid environments where internal services have their own DNS zone.

The ndots:5 Problem

The ndots:5 setting in /etc/resolv.conf is Kubernetes' most misunderstood DNS behavior. It means: if a name has fewer than 5 dots, treat it as a relative name and try all search domains before querying it as-is. The intent is to let you write short names like my-service or my-service.other-namespace and have them resolve through the search list.

The problem emerges with external domains. A name like api.github.com has only 2 dots — fewer than 5 — so the resolver doesn't query it directly. Instead, it tries the search domains first:

1. api.github.com.default.svc.cluster.local → NXDOMAIN
2. api.github.com.svc.cluster.local → NXDOMAIN
3. api.github.com.cluster.local → NXDOMAIN
4. api.github.com → ✔ resolved

That's four DNS queries instead of one for every external domain lookup. In applications that make heavy use of external APIs, this creates significant unnecessary DNS traffic and adds latency (typically 2–10ms per extra query). Multiply that across thousands of Pods making frequent HTTP calls and the load on CoreDNS becomes substantial.

Mitigations

You have several options to reduce the overhead, depending on how much control you have over the application and the Pod spec.

Option 1: Use FQDNs with a trailing dot. A trailing dot tells the resolver the name is absolute — skip the search list entirely. This requires changing application code or configuration.

# Instead of this (triggers search list):
api.github.com

# Use this (absolute, no search list):
api.github.com.

Option 2: Lower ndots in the Pod spec. Setting ndots:1 means any name with at least one dot (like api.github.com) is queried directly first. The trade-off is that cross-namespace short names like my-service.other-namespace require you to add .svc.cluster.local explicitly.

apiVersion: v1
kind: Pod
metadata:
  name: low-ndots-pod
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
  containers:
    - name: app
      image: my-app:1.0

Option 3: Use NodeLocal DNSCache. This DaemonSet runs a DNS caching agent on every node, reducing the latency and load caused by repeated queries. It doesn't eliminate extra queries, but it makes them much cheaper by serving cached NXDOMAIN responses locally.

Debugging DNS Issues

When Pods can't connect to Services or external endpoints, DNS is one of the first things to investigate. The fastest approach is to launch a temporary Pod with DNS tools and query CoreDNS directly.

Step 1: Launch a debug Pod

The busybox image includes nslookup, and the registry.k8s.io/e2e-test-images/jessie-dnsutils image includes both nslookup and dig. Use whichever is appropriate:

# Quick debug pod with nslookup
kubectl run dns-debug --rm -it --restart=Never \
  --image=busybox:1.36 -- sh

# Full debug pod with dig + nslookup
kubectl run dns-debug --rm -it --restart=Never \
  --image=registry.k8s.io/e2e-test-images/jessie-dnsutils -- bash

Step 2: Test cluster DNS resolution

# Resolve a Service in the same namespace
nslookup my-service

# Resolve a Service in another namespace
nslookup my-service.other-namespace.svc.cluster.local

# Resolve an external domain
nslookup google.com

# Use dig for detailed output (shows query path, TTL, response code)
dig my-service.default.svc.cluster.local

# Query CoreDNS directly by IP
dig @10.96.0.10 kubernetes.default.svc.cluster.local

Step 3: Check CoreDNS health

# Are CoreDNS pods running?
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs for errors
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Verify the kube-dns Service has endpoints
kubectl get endpoints kube-dns -n kube-system

Common DNS Problems and Causes

Network Policies — Microsegmentation in Kubernetes

By default, every Pod in a Kubernetes cluster can talk to every other Pod — across namespaces, across nodes, with zero restrictions. This is the "flat network" model baked into the Kubernetes networking specification. It makes getting started easy, but it is a serious security liability in any real environment.

Imagine a compromised frontend Pod freely reaching your database Pod, or a rogue workload in a shared cluster scanning every service on the network. Without network policies, there is nothing stopping lateral movement once an attacker gains a foothold in any Pod. NetworkPolicy resources give you microsegmentation — fine-grained firewall rules that restrict which Pods can communicate with which, on which ports, and in which direction.

flowchart LR
    subgraph "Default: No Network Policies"
        A[Pod A
frontend] <-->|allowed| B[Pod B
api]
        A <-->|allowed| C[Pod C
database]
        B <-->|allowed| C
        D[Pod D
untrusted] <-->|allowed| C
        D <-->|allowed| B
        D <-->|allowed| A
    end

    style D fill:#e74c3c,color:#fff,stroke:#c0392b
    style C fill:#2ecc71,color:#fff,stroke:#27ae60
    

In the diagram above, every Pod can reach every other Pod. The untrusted Pod has full access to the database — exactly the scenario Network Policies are designed to prevent.

The NetworkPolicy Resource

A NetworkPolicy is a namespaced resource that defines traffic rules for a group of Pods. It has three core building blocks: a pod selector that identifies which Pods the policy targets, ingress rules that control incoming traffic, and egress rules that control outgoing traffic. Here is the skeleton:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-frontend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database
      ports:
        - protocol: TCP
          port: 5432

This policy targets all Pods with app: api in the production namespace. It allows inbound traffic only from Pods labeled app: frontend on port 8080, and permits outbound traffic only to Pods labeled app: database on port 5432. All other traffic to and from the api Pods is denied.

spec.podSelector — Targeting Pods

The podSelector field uses standard label selectors to determine which Pods in the policy's namespace are affected. An empty selector (podSelector: {}) matches all Pods in the namespace — this is how you build default deny rules. A selector like matchLabels: {app: api, tier: backend} targets only Pods carrying both labels.

Ingress Rules — Controlling Inbound Traffic

The ingress array defines who is allowed to send traffic to the selected Pods. Each rule in the array is an independent allow rule. Within a single rule, the from array has three types of selectors that can be combined:

Here is the difference spelled out in YAML. The first example is an AND (both conditions must be true). The second is an OR (either condition allows traffic):

# AND — Pods labeled app:prometheus IN namespaces labeled team:monitoring
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            team: monitoring
        podSelector:
          matchLabels:
            app: prometheus

# OR — Pods labeled app:prometheus OR any Pod in team:monitoring namespaces
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            team: monitoring
      - podSelector:
          matchLabels:
            app: prometheus

Egress Rules — Controlling Outbound Traffic

Egress rules mirror ingress exactly, but use the to field instead of from. They support the same three selectors: podSelector, namespaceSelector, and ipBlock. Egress policies are critical for preventing compromised Pods from exfiltrating data or reaching external command-and-control servers.

egress:
  # Allow DNS resolution (critical — almost always needed)
  - to:
      - namespaceSelector: {}
    ports:
      - protocol: UDP
        port: 53
      - protocol: TCP
        port: 53
  # Allow outbound to the database
  - to:
      - podSelector:
          matchLabels:
            app: database
    ports:
      - protocol: TCP
        port: 5432

Port Specifications

Both ingress and egress rules support a ports array. Each entry specifies a protocol (TCP, UDP, or SCTP) and a port number or named port. You can also use endPort to specify a range. If you omit the ports field entirely, the rule applies to all ports.

ports:
  - protocol: TCP
    port: 8080            # single port
  - protocol: TCP
    port: 9090
    endPort: 9099         # port range 9090-9099
  - protocol: TCP
    port: http            # named port from Pod spec

How Policies Combine — The Additive Model

NetworkPolicies are additive. When multiple policies select the same Pod, the effective set of allowed connections is the union of all those policies. You cannot write a policy that removes access granted by another policy. This is by design — it makes the system predictable and prevents accidental lockouts from conflicting rules.

Here is how it works in practice: if Policy A allows ingress from the frontend on port 8080, and Policy B allows ingress from the monitoring namespace on port 9090, then both types of traffic are allowed. There is no priority or ordering between policies. The only way to restrict traffic is to not include it in any policy that selects those Pods — which is why starting with a deny-all baseline is so important.

Default Deny Policies

The strongest pattern for securing a namespace is to start with a deny-all policy and then layer on explicit allows. The moment any NetworkPolicy selects a Pod, all traffic not explicitly permitted by some policy is denied. A policy with an empty podSelector and empty rules achieves exactly this.

Default Deny All Ingress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}    # selects ALL Pods in namespace
  policyTypes:
    - Ingress        # no ingress rules = deny all inbound

This selects every Pod in the production namespace and declares Ingress as a policy type — but provides zero ingress rules. The result: all inbound traffic to every Pod in the namespace is blocked. Outbound traffic is unaffected because Egress is not listed in policyTypes.

Default Deny All Egress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Egress         # no egress rules = deny all outbound

Default Deny Both Directions

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

With this applied, no Pod in the namespace can send or receive any traffic until you explicitly add policies that allow it. This is the zero-trust starting point.

CNI Plugin Requirements

Here is the part that catches many teams off guard: NetworkPolicy resources are only enforced if your CNI plugin supports them. The Kubernetes API server will happily accept your NetworkPolicy manifests regardless — no errors, no warnings — but they are silently ignored if the CNI cannot enforce them.

Practical Patterns

Pattern 1: Isolating a Namespace

The most common starting point is locking down an entire namespace so that only Pods within it can communicate with each other. External namespaces cannot reach in, and Pods inside cannot reach out to other namespaces (except DNS).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-namespace
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Allow traffic only from within this namespace
    - from:
        - podSelector: {}
  egress:
    # Allow traffic within this namespace
    - to:
        - podSelector: {}
    # Allow DNS resolution
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Every Pod in tenant-a can reach other Pods in tenant-a and resolve DNS. Nothing else — no cross-namespace communication, no internet access.

Pattern 2: Deny All, Then Allow Specific Service-to-Service Communication

This is the zero-trust approach: start with a full deny, then create targeted policies for each legitimate communication path. The following example models a three-tier application where the frontend talks to the API, and the API talks to the database. Nothing else is permitted.

# 1. Deny all traffic in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
# 2. Allow DNS for all Pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
---
# 3. Frontend can receive external ingress, send to API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - ports:
        - protocol: TCP
          port: 80
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 8080
---
# 4. API receives from frontend, sends to database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database
      ports:
        - protocol: TCP
          port: 5432
---
# 5. Database receives from API only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: database-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 5432

flowchart LR
    EXT[External
Traffic] -->|":80"| FE[frontend]
    FE -->|":8080"| API[api]
    API -->|":5432"| DB[database]

    FE x--x|blocked| DB
    EXT x--x|blocked| DB
    EXT x--x|blocked| API

    style EXT fill:#95a5a6,color:#fff,stroke:#7f8c8d
    style FE fill:#3498db,color:#fff,stroke:#2980b9
    style API fill:#f39c12,color:#fff,stroke:#e67e22
    style DB fill:#2ecc71,color:#fff,stroke:#27ae60
    

This is five manifests, but the pattern is systematic. Each service gets exactly the connections it needs and nothing more. If the API is compromised, it can reach the database — but the database policy restricts it to port 5432, and the API cannot reach the internet or other namespaces.

Pattern 3: Allowing Cross-Namespace Monitoring

A common real-world requirement is allowing your monitoring stack (Prometheus, Grafana) to scrape metrics from application namespaces. Label the monitoring namespace and use a namespaceSelector:

# First, label the monitoring namespace:
# kubectl label namespace monitoring purpose=monitoring

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scraping
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              purpose: monitoring
          podSelector:
            matchLabels:
              app: prometheus
      ports:
        - protocol: TCP
          port: 9090
        - protocol: TCP
          port: 9091

Notice this uses the AND form — namespaceSelector and podSelector are in the same from entry. This means only Pods labeled app: prometheus within namespaces labeled purpose: monitoring can reach the metrics ports. Any other Pod in the monitoring namespace is still blocked.

Pattern 4: Restricting Egress to External CIDRs

Sometimes a Pod needs to reach an external service (a SaaS API, a managed database outside the cluster). Use ipBlock to allow traffic only to specific IP ranges:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-external-egress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: payment-service
  policyTypes:
    - Egress
  egress:
    # Allow DNS
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53
    # Allow Stripe API (example IPs)
    - to:
        - ipBlock:
            cidr: 54.187.174.169/32
        - ipBlock:
            cidr: 54.187.205.235/32
      ports:
        - protocol: TCP
          port: 443

The payment-service can reach exactly two external IP addresses on port 443 (HTTPS) and nothing else. Even if the service is compromised, data exfiltration to arbitrary internet hosts is blocked.

Debugging Network Policies

When traffic is unexpectedly blocked (or unexpectedly allowed), use this checklist:

• Verify your CNI supports NetworkPolicy. Run kubectl get pods -n kube-system and check which CNI is running.
• List policies affecting a Pod. Use kubectl get networkpolicy -n <namespace> and check each policy's podSelector against your Pod's labels.
• Check label accuracy. A typo in a label selector silently matches zero Pods. Run kubectl get pods --show-labels to verify.
• Test connectivity directly. Exec into a Pod and use curl, wget, or nc to test reachability: kubectl exec -it <pod> -- curl -m 3 <target>:<port>
• Remember DNS. If egress is restricted and DNS is not explicitly allowed, name resolution fails and every connection attempt by hostname fails with it.

Volumes — Attaching Storage to Pods

Every container in Kubernetes starts with a fresh, isolated filesystem layered from its image. When that container restarts — whether from a crash, a liveness probe failure, or a rolling update — everything written to that filesystem is gone. This is by design: containers are ephemeral. But most real applications need to persist data, share files between containers, or load configuration from external sources.

Kubernetes volumes solve this by providing a directory that is mounted into a container's filesystem and whose lifecycle is tied to the Pod (not the container). A volume outlives container restarts within the same Pod, and different volume types connect you to everything from temporary scratch space to cloud provider block storage.

The Ephemeral Filesystem Problem

To understand why volumes matter, consider what happens without them. A container writes log files, caches data, or stores user uploads to its local filesystem. When the kubelet restarts that container (say, after an OOMKill), the replacement container starts from a clean image layer. All written data is lost. If you have a sidecar container that needs to read those same log files, it cannot — each container has its own isolated filesystem root.

Volumes address both problems. They survive container restarts within a Pod, and they can be mounted into multiple containers simultaneously, enabling shared-storage patterns like the sidecar log collector or the adapter pattern.

Volume Lifecycle

A Kubernetes volume is declared at the Pod level (under spec.volumes) and mounted into one or more containers (under spec.containers[].volumeMounts). The critical rule: a volume's lifetime is tied to the Pod's lifetime. When the Pod is deleted, the volume is cleaned up — unless it points to external, persistent storage (which we cover in the next section on PersistentVolumes).

emptyDir — Temporary Shared Storage

An emptyDir volume starts as an empty directory when a Pod is assigned to a node. It exists for the lifetime of that Pod and is shared across all containers in the Pod. This is the workhorse volume for multi-container patterns: a main application container writes to the volume, and a sidecar reads from it.

By default, emptyDir is backed by the node's disk (whatever medium backs the node's filesystem). You can set medium: Memory to use a tmpfs RAM-backed filesystem instead, which is faster but counts against the container's memory limit and disappears on node reboot.

apiVersion: v1
kind: Pod
metadata:
  name: log-collector
spec:
  containers:
    - name: app
      image: nginx:1.27
      volumeMounts:
        - name: shared-logs
          mountPath: /var/log/nginx
    - name: sidecar
      image: busybox:1.36
      command: ["sh", "-c", "tail -F /logs/access.log"]
      volumeMounts:
        - name: shared-logs
          mountPath: /logs
          readOnly: true
  volumes:
    - name: shared-logs
      emptyDir: {}

In this example, Nginx writes access logs to /var/log/nginx. The sidecar container sees those same files at /logs because both mount the same emptyDir volume. The sidecar mounts it read-only since it only needs to tail the logs.

For a RAM-backed scratch volume with a size cap:

volumes:
  - name: scratch-space
    emptyDir:
      medium: Memory
      sizeLimit: 128Mi

hostPath — Access to the Node Filesystem

A hostPath volume mounts a file or directory from the host node's filesystem directly into the Pod. This gives containers access to node-level resources like Docker's socket, system logs, or device files. It is powerful but dangerous — you are breaking the isolation boundary between your Pod and the underlying node.

apiVersion: v1
kind: Pod
metadata:
  name: node-log-reader
spec:
  containers:
    - name: log-reader
      image: busybox:1.36
      command: ["sh", "-c", "cat /host-logs/syslog"]
      volumeMounts:
        - name: host-logs
          mountPath: /host-logs
          readOnly: true
  volumes:
    - name: host-logs
      hostPath:
        path: /var/log
        type: Directory

The type field adds safety checks before the mount. Common values include Directory (must already exist as a directory), File (must already exist as a file), DirectoryOrCreate (create if missing), and "" (empty string, no checks — the default).

configMap and secret — Mount Configuration as Files

ConfigMaps and Secrets are Kubernetes API objects that store key-value data. When you mount them as volumes, each key becomes a file in the mounted directory, with the value as the file content. This is how you inject configuration files, TLS certificates, or credential files into containers without baking them into the image.

ConfigMap Volume

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    server {
      listen 80;
      location / {
        root /usr/share/nginx/html;
      }
    }
  extra-settings.conf: |
    gzip on;
---
apiVersion: v1
kind: Pod
metadata:
  name: web-server
spec:
  containers:
    - name: nginx
      image: nginx:1.27
      volumeMounts:
        - name: config-vol
          mountPath: /etc/nginx/conf.d
  volumes:
    - name: config-vol
      configMap:
        name: nginx-config

After mounting, the container's /etc/nginx/conf.d directory contains two files: nginx.conf and extra-settings.conf. If you only need specific keys, use the items field to select them and optionally remap filenames:

volumes:
  - name: config-vol
    configMap:
      name: nginx-config
      items:
        - key: nginx.conf
          path: default.conf

This mounts only the nginx.conf key, but names the resulting file default.conf inside the mount directory.

Secret Volume

Secret volumes work identically to ConfigMap volumes, except the data comes from a Secret object. Kubernetes mounts Secret files with 0644 permissions by default; you can tighten this with defaultMode. The files are backed by a tmpfs in memory — they are never written to the node's disk.

apiVersion: v1
kind: Pod
metadata:
  name: tls-app
spec:
  containers:
    - name: app
      image: my-app:2.1
      volumeMounts:
        - name: tls-certs
          mountPath: /etc/tls
          readOnly: true
  volumes:
    - name: tls-certs
      secret:
        secretName: app-tls
        defaultMode: 0400

The Secret app-tls might contain keys tls.crt and tls.key. After mounting, the container finds them at /etc/tls/tls.crt and /etc/tls/tls.key, both with restrictive 0400 permissions (owner read-only).

downwardAPI — Expose Pod Metadata as Files

The downwardAPI volume projects Pod metadata — labels, annotations, resource limits, the Pod name, namespace, and node name — into files inside the container. This is useful when your application needs self-awareness without querying the Kubernetes API directly.

apiVersion: v1
kind: Pod
metadata:
  name: self-aware-app
  labels:
    app: backend
    version: v2
  annotations:
    owner: platform-team
spec:
  containers:
    - name: app
      image: my-app:2.1
      volumeMounts:
        - name: pod-info
          mountPath: /etc/podinfo
  volumes:
    - name: pod-info
      downwardAPI:
        items:
          - path: labels
            fieldRef:
              fieldPath: metadata.labels
          - path: annotations
            fieldRef:
              fieldPath: metadata.annotations
          - path: cpu-limit
            resourceFieldRef:
              containerName: app
              resource: limits.cpu

Inside the container, /etc/podinfo/labels contains the content app="backend"\nversion="v2". The cpu-limit file contains the numeric CPU limit. Labels and annotations are kept in sync — if an annotation changes on the running Pod, the mounted file updates automatically (with a short delay).

projected — Combine Multiple Sources

A projected volume merges multiple volume sources — Secrets, ConfigMaps, downwardAPI fields, and serviceAccountToken — into a single mount point. Without projected volumes, you would need separate volume declarations and separate mount paths for each source. This quickly becomes unwieldy when a container needs a TLS cert from a Secret, a config file from a ConfigMap, and a service account token all in the same directory.

apiVersion: v1
kind: Pod
metadata:
  name: combined-config
spec:
  containers:
    - name: app
      image: my-app:2.1
      volumeMounts:
        - name: all-config
          mountPath: /etc/app-config
          readOnly: true
  volumes:
    - name: all-config
      projected:
        sources:
          - configMap:
              name: app-settings
              items:
                - key: app.yaml
                  path: app.yaml
          - secret:
              name: app-tls
              items:
                - key: tls.crt
                  path: certs/tls.crt
                - key: tls.key
                  path: certs/tls.key
          - downwardAPI:
              items:
                - path: pod-name
                  fieldRef:
                    fieldPath: metadata.name
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600
              audience: vault

The resulting directory at /etc/app-config contains app.yaml (from the ConfigMap), certs/tls.crt and certs/tls.key (from the Secret), pod-name (from the downward API), and token (a bound service account token with a 1-hour expiry). The serviceAccountToken source is particularly useful — it generates short-lived, audience-scoped tokens, which is the preferred alternative to the long-lived tokens from the default service account Secret.

subPath — Mount a Single File Without Hiding a Directory

When you mount a volume to a container path, it replaces the entire directory at that path. If the container image already has files in /etc/nginx/conf.d and you mount a ConfigMap there, the original files disappear. The subPath field solves this by mounting a single file (or subdirectory) from the volume into the target path, leaving the rest of the directory untouched.

apiVersion: v1
kind: Pod
metadata:
  name: subpath-demo
spec:
  containers:
    - name: nginx
      image: nginx:1.27
      volumeMounts:
        - name: custom-config
          mountPath: /etc/nginx/conf.d/custom.conf
          subPath: custom.conf
  volumes:
    - name: custom-config
      configMap:
        name: nginx-custom

Now only custom.conf is placed at /etc/nginx/conf.d/custom.conf. The image's default default.conf and any other files in that directory remain intact.

volumeMounts Options Reference

Beyond mountPath and subPath, the volumeMounts field supports several options that control how the volume behaves inside the container.

Volume Type Comparison

Choosing the right volume type depends on what data you need and how long it should survive. This table summarizes the core Pod-level volume types at a glance.

* ConfigMap and Secret volumes are mounted read-only by default since Kubernetes 1.19+ when the Immutable feature is used. Without that, writes technically succeed in the container but are not persisted back to the API object.

Putting It Together — A Complete Multi-Volume Pod

Real-world Pods often combine several volume types. Here is a complete example: a web application Pod that loads its config from a ConfigMap, its TLS certificates from a Secret, writes temporary cache data to an emptyDir, and exposes Pod labels to the application.

apiVersion: v1
kind: Pod
metadata:
  name: production-web
  labels:
    app: web
    tier: frontend
spec:
  containers:
    - name: web
      image: my-web-app:3.4
      ports:
        - containerPort: 8443
      volumeMounts:
        - name: app-config
          mountPath: /etc/app/config.yaml
          subPath: config.yaml
          readOnly: true
        - name: tls
          mountPath: /etc/tls
          readOnly: true
        - name: cache
          mountPath: /tmp/cache
        - name: pod-meta
          mountPath: /etc/podinfo
          readOnly: true
      resources:
        limits:
          memory: 256Mi
          cpu: 500m
  volumes:
    - name: app-config
      configMap:
        name: web-app-config
    - name: tls
      secret:
        secretName: web-tls-cert
        defaultMode: 0400
    - name: cache
      emptyDir:
        sizeLimit: 100Mi
    - name: pod-meta
      downwardAPI:
        items:
          - path: labels
            fieldRef:
              fieldPath: metadata.labels

This pattern — config from a ConfigMap, secrets from a Secret, ephemeral cache from emptyDir, and metadata from the downward API — covers the majority of volume needs for stateless applications. When you need storage that survives Pod deletion, that is when you reach for PersistentVolumes, covered next.

PersistentVolumes and PersistentVolumeClaims

Containers are ephemeral, but data often is not. A database, a file upload service, or a message queue all need storage that survives Pod restarts, rescheduling, and even node failures. Kubernetes solves this with a two-object model: PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). This separation cleanly divides infrastructure provisioning from application consumption.

The Separation of Concerns

A PersistentVolume (PV) is a cluster-level resource representing a piece of physical or cloud storage — an NFS export, an AWS EBS volume, a GCE Persistent Disk, or a local SSD. PVs are created by cluster administrators (or dynamically by a provisioner) and exist independently of any Pod. Think of a PV as the actual hard drive sitting in a rack.

A PersistentVolumeClaim (PVC) is a namespaced request for storage made by a user or workload. It specifies how much storage is needed and how it will be accessed, without caring about where the storage comes from. The PVC is the purchase order; the PV is the inventory item that fulfills it.

The PV Lifecycle

A PersistentVolume moves through four distinct phases from creation to cleanup. Understanding this lifecycle is critical for debugging storage issues and planning capacity.

stateDiagram-v2
    [*] --> Available : Provisioning (Static or Dynamic)
    Available --> Bound : PVC matches PV
    Bound --> Released : PVC is deleted
    Released --> Available : Reclaim policy = Recycle (deprecated)
    Released --> [*] : Reclaim policy = Delete
    Released --> Released : Reclaim policy = Retain (manual cleanup)
    

1. Provisioning

Before a PV can be used, it must exist. There are two provisioning strategies, and most production clusters use both.

Static provisioning means a cluster administrator creates PV objects by hand, each pointing to a specific backing storage resource. This is common with on-premises infrastructure — NFS servers, iSCSI targets, or local disks — where automated provisioning is not available.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-01
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 10.0.0.5
    path: /exports/data

Dynamic provisioning eliminates the need for pre-created PVs. When a PVC references a StorageClass, Kubernetes automatically provisions a matching PV through the class's provisioner plugin. This is the default model on cloud providers (AWS, GCP, Azure) and CSI-based storage systems.

2. Binding

When a PVC is created, the control plane searches for an available PV that satisfies the claim's requirements: sufficient capacity, compatible access modes, matching StorageClass, and any label selectors. If a match is found, the PVC and PV are bound in a one-to-one relationship. This binding is exclusive — once a PV is bound to a PVC, no other PVC can use it, even if the PV has more capacity than the claim requested.

If no matching PV exists and no StorageClass can dynamically provision one, the PVC remains in a Pending state indefinitely until a suitable PV becomes available.

3. Using

Once bound, a Pod can mount the PVC as a volume. The cluster looks up the PVC, finds the bound PV, and mounts the underlying storage into the Pod's container at the specified path. Multiple Pods can use the same PVC simultaneously if the access mode permits it (e.g., ReadWriteMany).

apiVersion: v1
kind: Pod
metadata:
  name: app-server
spec:
  containers:
    - name: app
      image: myapp:3.2
      volumeMounts:
        - mountPath: /var/data
          name: data-volume
  volumes:
    - name: data-volume
      persistentVolumeClaim:
        claimName: app-data-pvc

4. Reclaiming

When a PVC is deleted, its bound PV enters the Released phase. What happens next depends on the PV's reclaim policy. This is where storage cleanup strategy is configured, and getting it wrong can mean either data loss or orphaned cloud volumes accumulating cost.

Access Modes

Access modes define how a volume can be mounted by nodes. They are constraints on node-level access, not Pod-level — a volume with ReadWriteOnce can be mounted by multiple Pods, but only if they all run on the same node.

Not every storage backend supports every access mode. Block storage (EBS, GCE PD, Azure Disk) typically supports only RWO. File-based storage (NFS, EFS, CephFS) can support RWX. Always check your storage provider's documentation.

Reclaim Policies

The reclaim policy determines what happens to a PV (and its underlying storage) after its bound PVC is deleted. This is set on the PV, not the PVC.

Dynamically provisioned PVs inherit the reclaim policy from their StorageClass. The default for most cloud StorageClasses is Delete. If you are running stateful workloads like databases, you should either change the StorageClass default to Retain or patch individual PVs after creation.

# Patch an existing PV to Retain so deleting the PVC won't destroy data
kubectl patch pv my-database-pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Volume Binding Modes

Volume binding mode controls when a PV is bound to a PVC. This is configured on the StorageClass, not on the PV or PVC directly. The choice has real implications for scheduling and data locality.

Immediate (the default) — The PV is provisioned and bound as soon as the PVC is created, regardless of whether any Pod has requested it yet. This works fine for storage that is accessible from any node (like network-attached storage). But it can cause problems with topology-constrained storage: if an EBS volume is provisioned in us-east-1a but the Pod gets scheduled to a node in us-east-1b, the Pod will be stuck in Pending.

WaitForFirstConsumer — The PV binding and provisioning are delayed until a Pod that uses the PVC is scheduled. The scheduler considers the Pod's node assignment first, then provisions storage in the same topology zone. This is the recommended mode for zone-constrained block storage like EBS, GCE PD, and Azure Disk.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3
  iops: "4000"

How PV-PVC Matching Works

When a PVC is created with no specific volumeName, the control plane runs a matching algorithm to find the best available PV. The criteria are evaluated in this order:

1. StorageClass — The PVC's storageClassName must exactly match the PV's storageClassName. A PVC with storageClassName: "" (empty string) only matches PVs with no class. A PVC with no storageClassName field uses the cluster's default StorageClass.
2. Access Modes — The PV must support at least the access modes requested by the PVC. A PV offering [RWO, ROX] satisfies a PVC requesting [RWO].
3. Capacity — The PV's capacity must be greater than or equal to the PVC's requested storage. Kubernetes picks the smallest PV that satisfies the claim to minimize waste.
4. Label Selectors — If the PVC defines a selector with matchLabels or matchExpressions, only PVs matching those labels are considered.
5. Volume Name — If the PVC specifies volumeName, it skips all other matching logic and binds directly to that specific PV (assuming access modes and capacity are compatible).

Putting It All Together: Static Provisioning Example

The following example shows the complete workflow: an admin creates a PV backed by NFS, a developer creates a PVC that matches it, and a Deployment mounts the claim.

# 1. Admin creates the PersistentVolume
apiVersion: v1
kind: PersistentVolume
metadata:
  name: shared-nfs-pv
  labels:
    environment: production
    tier: storage
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""          # Empty string = no dynamic provisioning
  nfs:
    server: 10.0.0.5
    path: /exports/shared-data

# 2. Developer creates a PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data-pvc
  namespace: web-app
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  storageClassName: ""          # Must match the PV's storageClassName
  selector:
    matchLabels:
      environment: production

# 3. Deployment mounts the PVC
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          volumeMounts:
            - mountPath: /usr/share/nginx/html
              name: shared-content
      volumes:
        - name: shared-content
          persistentVolumeClaim:
            claimName: shared-data-pvc

Dynamic Provisioning Example

With dynamic provisioning, you skip the PV creation entirely. The PVC references a StorageClass, and Kubernetes creates the PV automatically. This is the standard approach on cloud-managed clusters.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: database
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: fast-ssd     # References the StorageClass provisioner

Once this PVC is applied and a Pod referencing it is scheduled, the fast-ssd StorageClass provisioner creates a 20Gi volume in the same availability zone as the node (because we set WaitForFirstConsumer earlier). A corresponding PV object is automatically created and bound to this PVC.

Inspecting PVs and PVCs

When debugging storage issues, these commands show you the current state and help identify mismatches.

# List all PVs with their status, capacity, access modes, and bound claim
kubectl get pv

# List PVCs in a namespace with their bound PV
kubectl get pvc -n database

# Describe a PVC to see events — useful when a PVC is stuck in Pending
kubectl describe pvc postgres-data -n database

# Check why a PV is in Released state and not reusable
kubectl get pv shared-nfs-pv -o yaml | grep -A 5 claimRef

A common gotcha: when a PV has reclaim policy Retain and its PVC is deleted, the PV moves to Released but still holds a claimRef pointing to the old PVC. No new PVC can bind to it until you manually clear that reference:

# Remove the stale claimRef so the PV becomes Available again
kubectl patch pv shared-nfs-pv --type json \
  -p '[{"op": "remove", "path": "/spec/claimRef"}]'

StorageClasses and Dynamic Provisioning

In the previous section, you created PersistentVolumes by hand and then matched them with PersistentVolumeClaims. That workflow is fine for a handful of volumes, but it collapses under real-world conditions. If you have 50 microservices each needing their own volume — across dev, staging, and production — someone has to manually create and manage 150+ PV manifests. Worse, those PVs must be pre-provisioned in the cloud provider before a workload can claim them.

StorageClasses solve this by letting you declare what kind of storage you want. When a PVC references a StorageClass, Kubernetes automatically calls the appropriate provisioner plugin to create the underlying volume, wraps it in a PV object, and binds it to the PVC — all without human intervention. This is dynamic provisioning, and it is how virtually every production cluster manages storage.

The StorageClass Spec

A StorageClass is a cluster-scoped resource (not namespaced) that acts as a template for volume creation. It tells Kubernetes which provisioner to call and what parameters to pass. Here is the anatomy of a StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com        # Which plugin creates the volume
parameters:                          # Provider-specific settings
  type: gp3
  fsType: ext4
  iopsPerGB: "50"
reclaimPolicy: Delete                # What happens when PVC is deleted
volumeBindingMode: WaitForFirstConsumer  # When to provision
allowVolumeExpansion: true           # Can PVCs grow after creation?

Each field controls a different aspect of the provisioning lifecycle. Let's walk through them.

provisioner

The provisioner field identifies the volume plugin responsible for creating and deleting the actual storage backend. Every cloud provider and storage system has its own provisioner. The older "in-tree" provisioners (like kubernetes.io/aws-ebs) are built into Kubernetes itself, but they are deprecated. Modern clusters use out-of-tree CSI drivers instead.

parameters

The parameters map is passed directly to the provisioner. Its contents are entirely provider-specific — Kubernetes does not validate them. Here are common parameters for the major cloud providers:

All parameter values must be strings — even numeric ones. Writing iopsPerGB: 50 without quotes will cause YAML to pass an integer, which some provisioners reject. Always quote: iopsPerGB: "50".

reclaimPolicy

The reclaimPolicy determines what happens to the underlying volume when the PVC is deleted. There are two options:

• Delete (default): The PV and its backing storage are destroyed. This is the right choice for ephemeral or easily-recreatable data.
• Retain: The PV is kept and its status changes to Released. The actual cloud volume is not deleted. You must manually reclaim or clean it up. Use this for databases and anything you cannot afford to lose.

Note that Recycle (which ran rm -rf on the volume) is deprecated and unsupported by CSI drivers. If you need similar behavior, use a Delete policy and rely on backups.

volumeBindingMode

This field controls when the volume is actually provisioned and bound. It has a significant impact on scheduling and is the most commonly misconfigured StorageClass field.

allowVolumeExpansion

When set to true, you can increase the size of an existing PVC by editing its spec.resources.requests.storage field. The CSI driver will expand the underlying volume and, if needed, resize the filesystem. Most cloud CSI drivers support this. You cannot shrink a volume — only grow it.

The Default StorageClass

When a PVC does not specify a storageClassName, Kubernetes assigns it the default StorageClass. A StorageClass is marked as default using the storageclass.kubernetes.io/is-default-class annotation. Most managed Kubernetes clusters (EKS, GKE, AKS) come with a default StorageClass pre-configured.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

You can check which StorageClass is the default with kubectl get sc. The default will have (default) next to its name:

$ kubectl get storageclass
NAME                 PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE       AGE
fast-ssd             ebs.csi.aws.com   Delete          WaitForFirstConsumer    5d
standard (default)   ebs.csi.aws.com   Delete          WaitForFirstConsumer    30d

If you have more than one StorageClass annotated as default, the behavior is undefined — some admission controllers will reject PVCs, while others will pick one arbitrarily. Keep exactly one default per cluster.

How Dynamic Provisioning Works End-to-End

Understanding the full lifecycle clarifies what Kubernetes does behind the scenes when you create a PVC. Here is the sequence:

1. You create a PVC that references a StorageClass (either explicitly via storageClassName or implicitly via the default).
2. The PVC enters Pending state. If the binding mode is WaitForFirstConsumer, it stays Pending until a Pod claims it.
3. A Pod is scheduled that mounts the PVC. The scheduler picks a node, taking storage topology into account.
4. The provisioner plugin fires. It calls the cloud provider API (e.g., ec2:CreateVolume) to create a volume in the appropriate zone with the specified parameters.
5. Kubernetes creates a PV object that represents the newly provisioned volume. The PV's spec.claimRef is set to the PVC.
6. The PVC is bound to the PV. Both transition to Bound status.
7. The kubelet mounts the volume into the Pod's container at the specified mountPath.

When the PVC is deleted, the process runs in reverse: the volume is unmounted, the PV is removed, and (if reclaimPolicy: Delete) the cloud volume is destroyed.

Example: AWS EBS with the CSI Driver

This is the most common setup on EKS. You define a StorageClass for gp3 SSD volumes and create a PVC that triggers dynamic provisioning.

# storageclass-aws.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
# pvc-aws.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-gp3
  resources:
    requests:
      storage: 20Gi

Apply both and watch the PVC wait for a consumer:

$ kubectl apply -f storageclass-aws.yaml -f pvc-aws.yaml

$ kubectl get pvc app-data
NAME       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
app-data   Pending                                      ebs-gp3        5s
# Pending — waiting for a Pod to be scheduled (WaitForFirstConsumer)

$ kubectl run test --image=nginx --overrides='{
  "spec": {"containers": [{"name": "nginx", "image": "nginx",
    "volumeMounts": [{"name": "data", "mountPath": "/data"}]}],
    "volumes": [{"name": "data",
      "persistentVolumeClaim": {"claimName": "app-data"}}]}}'

$ kubectl get pvc app-data
NAME       STATUS   VOLUME                                     CAPACITY   STORAGECLASS   AGE
app-data   Bound    pvc-a1b2c3d4-5678-90ef-ghij-klmnopqrstuv   20Gi       ebs-gp3        45s

The PVC transitions from Pending to Bound once the Pod is scheduled. A PV named pvc-a1b2c3d4-... was automatically created by the EBS CSI driver, backed by a real EBS volume in the same availability zone as the node.

Example: GCP Persistent Disk with the CSI Driver

On GKE, the pattern is identical — only the provisioner name and parameters differ.

# storageclass-gcp.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pd-ssd
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
# pvc-gcp.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-storage
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: pd-ssd
  resources:
    requests:
      storage: 50Gi

For workloads that need replication across zones (e.g., a regional GKE cluster), add the replication-type parameter:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pd-ssd-regional
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  replication-type: regional-pd
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Example: Local Path Provisioner for Development

Cloud provisioners don't work on local clusters like kind, minikube, or k3s. The Rancher Local Path Provisioner fills this gap by creating volumes as directories on the node's filesystem. It ships by default with k3s and can be installed on any cluster.

# This StorageClass comes pre-installed on k3s clusters
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

PVCs referencing local-path work exactly like cloud-backed PVCs. The provisioner creates a directory under /opt/local-path-provisioner on the node, and the PV points to that host path. This gives you a fully functional dynamic provisioning loop for development and CI without any cloud infrastructure.

# Install local-path-provisioner on kind or minikube
$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.28/deploy/local-path-storage.yaml

# Verify it's running
$ kubectl -n local-path-storage get pod
NAME                                      READY   STATUS    AGE
local-path-provisioner-7745554f7f-k8x2q   1/1     Running   30s

Expanding a Volume After Creation

If your StorageClass has allowVolumeExpansion: true, you can grow a PVC in place. Edit the PVC and increase the storage request — the CSI driver handles the rest.

# Expand from 20Gi to 50Gi
$ kubectl patch pvc app-data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

# Watch the resize progress
$ kubectl get pvc app-data -o jsonpath='{.status.conditions[*].type}'
FileSystemResizePending

# After a moment (once the Pod remounts or the filesystem is resized online):
$ kubectl get pvc app-data
NAME       STATUS   VOLUME          CAPACITY   STORAGECLASS   AGE
app-data   Bound    pvc-a1b2c3d4    50Gi       ebs-gp3        2h

Some CSI drivers resize the filesystem online (while the Pod is running). Others require the Pod to be restarted for the resize to take effect. Check your driver's documentation. AWS EBS CSI and GCP PD CSI both support online expansion.

CSI Drivers and Storage Patterns for Production

Before Kubernetes v1.13, every storage backend — AWS EBS, GCE PD, Cinder, Ceph — was compiled directly into the Kubernetes codebase as an "in-tree" volume plugin. This meant that adding a new storage driver or fixing a bug required a full Kubernetes release. CSI (Container Storage Interface) decouples storage from core Kubernetes, letting vendors ship and update their own drivers on their own schedule.

CSI is not Kubernetes-specific. It is a cross-platform specification (also adopted by Mesos and Cloud Foundry) that defines a standard gRPC interface between a container orchestrator and a storage provider. In Kubernetes, CSI drivers run as a set of sidecar containers alongside a driver-specific plugin, all deployed as regular Pods.

Why In-Tree Plugins Had to Go

The in-tree model created three compounding problems. First, release coupling: a storage vendor could not ship a fix without waiting for the next Kubernetes minor release cycle (roughly every four months). Second, binary bloat: every kubelet and controller-manager binary contained code for every supported storage backend, even the ones that cluster never used. Third, privileged access: storage code ran inside core Kubernetes components, which meant a bug in a volume plugin could crash the kubelet or controller-manager.

CSI solves all three. Drivers are independently versioned container images. They run in their own Pods with scoped privileges. And the core Kubernetes codebase no longer needs to know anything about the underlying storage technology — it just speaks the CSI gRPC protocol.

CSI Architecture

A CSI driver deployment consists of two parts: a controller component (typically a Deployment or StatefulSet with one replica) that handles cluster-level operations like provisioning and attaching volumes, and a node component (a DaemonSet) that runs on every node to handle mounting and unmounting. Both parts are composed of Kubernetes-maintained sidecar containers plus the vendor-specific CSI driver container.

flowchart LR
    PVC["PVC Created"] --> EP["external-provisioner"]
    EP -->|"CreateVolume gRPC"| Driver["CSI Driver\n(Controller)"]
    Driver --> Backend["Storage Backend\n(EBS, Ceph, NFS...)"]
    Backend -->|"Volume Ready"| EA["external-attacher"]
    EA -->|"ControllerPublishVolume\ngRPC"| Driver
    Driver --> Attach["Volume Attached\nto Node"]
    Attach --> Kubelet["kubelet"]
    Kubelet -->|"NodeStageVolume\nNodePublishVolume"| NodeDriver["CSI Driver\n(Node DaemonSet)"]
    NodeDriver --> Mount["Volume Mounted\nin Container"]

    style PVC fill:#e0e7ff,stroke:#4f46e5,color:#1e1b4b
    style Backend fill:#fef3c7,stroke:#d97706,color:#78350f
    style Mount fill:#d1fae5,stroke:#059669,color:#064e3b
    

The diagram above shows the full lifecycle of a dynamically provisioned volume. The external sidecars watch for Kubernetes API events (new PVC, new VolumeAttachment) and translate them into gRPC calls to the CSI driver. The driver then talks to the actual storage backend.

CSI Sidecar Containers

Kubernetes maintains a set of standard sidecar containers that handle the orchestrator-side logic. These run alongside the vendor's CSI driver container within the same Pod. Here is what each one does:

How CSI Drivers Are Deployed

Most CSI drivers ship as Helm charts or kustomize manifests. Under the hood, the deployment always follows the same two-part pattern: a controller Deployment and a node DaemonSet. Here is a simplified view of the node DaemonSet for the AWS EBS CSI driver:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ebs-csi-node
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ebs-csi-node
  template:
    metadata:
      labels:
        app: ebs-csi-node
    spec:
      containers:
        - name: ebs-plugin
          image: public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.28.0
          volumeMounts:
            - name: kubelet-dir
              mountPath: /var/lib/kubelet
              mountPropagation: Bidirectional
            - name: plugin-dir
              mountPath: /csi
        - name: node-driver-registrar
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0
          args:
            - "--csi-address=/csi/csi.sock"
            - "--kubelet-registration-path=/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
      volumes:
        - name: kubelet-dir
          hostPath:
            path: /var/lib/kubelet
            type: Directory
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/ebs.csi.aws.com/
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory

Notice the mountPropagation: Bidirectional on the kubelet directory — this is critical. It allows the CSI driver to mount volumes inside its own mount namespace and have those mounts propagate to the kubelet, which then bind-mounts them into application containers. The node-driver-registrar sidecar registers the driver's Unix socket with the kubelet so the kubelet knows how to reach it.

Volume Snapshots

Volume snapshots bring point-in-time copy capabilities to Kubernetes. They are modeled as three API resources that mirror the PV/PVC pattern: VolumeSnapshotClass defines how to snapshot (which CSI driver, what parameters), VolumeSnapshot is the user-facing request (like a PVC), and VolumeSnapshotContent is the actual snapshot object bound to the backend (like a PV).

Volume snapshots require the CSI driver to implement the snapshot capability, and you must install the snapshot CRDs and the snapshot-controller separately — they are not part of core Kubernetes. Most managed Kubernetes offerings (EKS, GKE, AKS) pre-install these.

# 1. Define a VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshot-class
driver: ebs.csi.aws.com
deletionPolicy: Retain
---
# 2. Take a snapshot of an existing PVC
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: db-snapshot-2024-01-15
spec:
  volumeSnapshotClassName: ebs-snapshot-class
  source:
    persistentVolumeClaimName: postgres-data

The deletionPolicy controls what happens to the underlying snapshot in the storage backend when you delete the VolumeSnapshot object. Retain keeps the backend snapshot (safer for backups), while Delete removes it. Always use Retain for any snapshot that serves as a backup.

Restoring from a Snapshot

To restore, you create a new PVC with a dataSource pointing to the snapshot. Kubernetes provisions a new volume pre-populated with the snapshot data:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-restored
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3-encrypted
  resources:
    requests:
      storage: 100Gi
  dataSource:
    name: db-snapshot-2024-01-15
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Volume Cloning

Volume cloning creates a duplicate of an existing PVC without going through a snapshot intermediate. The clone is a new, independent volume with the same data as the source at the time of the clone request. It is useful for spinning up test environments from production data or creating read replicas.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-clone
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3-encrypted
  resources:
    requests:
      storage: 100Gi
  dataSource:
    name: postgres-data        # source PVC
    kind: PersistentVolumeClaim

The source and clone must be in the same namespace and use the same StorageClass. The clone size must be equal to or larger than the source. Some CSI drivers (like AWS EBS CSI) implement cloning via an internal snapshot-and-restore, so it can take time proportional to volume size.

Volume Expansion

CSI drivers that support the ControllerExpandVolume and NodeExpandVolume RPCs allow you to grow PVCs without downtime. The StorageClass must set allowVolumeExpansion: true, and then you simply patch the PVC's spec.resources.requests.storage to a larger value:

# Expand a PVC from 100Gi to 200Gi
kubectl patch pvc postgres-data -p \
  '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

# Monitor the resize progress
kubectl get pvc postgres-data -o jsonpath='{.status.conditions[*]}'

Expansion happens in two phases: the controller expands the underlying volume in the storage backend, then the node resizes the filesystem when the volume is next mounted (or immediately, for online expansion). You can track progress via the FileSystemResizePending condition on the PVC.

Production Storage Patterns

ReadWriteMany with NFS, CephFS, or EFS

Most block storage (EBS, Azure Disk, GCE PD) only supports ReadWriteOnce — a single node can mount the volume for read-write at a time. When multiple Pods across different nodes need to read and write the same data (shared uploads, CMS content, ML training datasets), you need a storage backend that supports ReadWriteMany (RWX).

# StorageClass for AWS EFS (ReadWriteMany)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap       # uses EFS Access Points
  fileSystemId: fs-0a1b2c3d4e5f
  directoryPerms: "700"
  basePath: "/dynamic_provisioning"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-uploads
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 50Gi

Block Storage vs. Filesystem Storage

CSI supports two volume modes: Filesystem (the default) and Block. With Filesystem mode, the CSI driver creates a filesystem (ext4, xfs) on the volume and mounts it as a directory. With Block mode, the raw block device is exposed directly to the container as a device file at the specified devicePath.

Block mode is used by databases that manage their own storage layout (like certain configurations of Oracle, or high-performance key-value stores), and by applications that need direct I/O without filesystem overhead. Most workloads should use Filesystem mode.

# Raw block volume PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block              # not Filesystem
  storageClassName: gp3-encrypted
  resources:
    requests:
      storage: 50Gi
---
# Pod consuming a raw block device
apiVersion: v1
kind: Pod
metadata:
  name: block-consumer
spec:
  containers:
    - name: app
      image: myapp:latest
      volumeDevices:             # not volumeMounts
        - name: data
          devicePath: /dev/xvda
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: raw-block-pvc

Ephemeral CSI Volumes

Some CSI drivers support ephemeral inline volumes — volumes defined directly in the Pod spec rather than through a PVC. These volumes have the same lifecycle as the Pod: they are created when the Pod starts and deleted when the Pod is removed. This pattern is ideal for injecting short-lived secrets, certificates, or identity tokens from external systems (e.g., HashiCorp Vault via the Secrets Store CSI Driver).

apiVersion: v1
kind: Pod
metadata:
  name: app-with-secrets
spec:
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: vault-secrets
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: vault-secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: vault-db-creds

Generic Ephemeral Volumes

Generic ephemeral volumes (GA since Kubernetes 1.23) combine the lifecycle of ephemeral volumes with the full power of PVCs. You embed a PVC template directly inside the Pod spec. Kubernetes creates a real PVC for each Pod, provisions a volume through the normal StorageClass flow, and automatically deletes the PVC when the Pod terminates.

This is powerful for workloads that need fast scratch space backed by real block storage (not just emptyDir on the node's disk). Think ML training jobs that need 500 GiB of NVMe-backed scratch, or CI runners that need isolated high-IOPS build volumes.

apiVersion: v1
kind: Pod
metadata:
  name: ml-training-job
spec:
  containers:
    - name: trainer
      image: ml-framework:latest
      volumeMounts:
        - name: scratch
          mountPath: /scratch
  volumes:
    - name: scratch
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: ["ReadWriteOnce"]
            storageClassName: io2-high-iops
            resources:
              requests:
                storage: 500Gi

Storage Best Practices for Production

Capacity Planning

Do not wait for PersistentVolume fullness alerts to plan capacity. Set up monitoring that tracks volume utilization percentage (available via the kubelet's /metrics endpoint as kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes). Alert at 70% utilization to give yourself time to expand. Combine this with volume expansion so that responding to an alert is a single kubectl patch rather than a data migration.

IOPS and Throughput Requirements

Storage performance problems are the number one cause of unexplained application slowness in Kubernetes. Cloud block storage IOPS scales with volume size (e.g., AWS gp3 provides a baseline of 3,000 IOPS for any size, but io2 scales up to 64,000). Match your StorageClass to your workload profile:

Backup Strategies with Snapshots

Snapshots are not backups by themselves — they often live in the same storage system as the original volume (e.g., EBS snapshots are in the same region). A production backup strategy should include: (1) scheduled VolumeSnapshots via a CronJob or a tool like Velero, (2) cross-region copy of snapshots for disaster recovery, and (3) periodic restore tests to verify snapshot integrity. Here is a minimal CronJob-based snapshot approach:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-snapshot-daily
spec:
  schedule: "0 2 * * *"           # 2 AM daily
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: snapshot-creator
          containers:
            - name: snapshot
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  SNAP_NAME="db-snap-$(date +%Y%m%d-%H%M%S)"
                  cat <<EOF | kubectl apply -f -
                  apiVersion: snapshot.storage.k8s.io/v1
                  kind: VolumeSnapshot
                  metadata:
                    name: ${SNAP_NAME}
                  spec:
                    volumeSnapshotClassName: ebs-snapshot-class
                    source:
                      persistentVolumeClaimName: postgres-data
                  EOF
                  echo "Created snapshot: ${SNAP_NAME}"
          restartPolicy: OnFailure

Cross-AZ Considerations

Block storage volumes (EBS, Azure Disk, GCE PD) are tied to a single availability zone. If a Pod is scheduled on a node in us-east-1a but its PV lives in us-east-1b, the volume attach will fail. Kubernetes handles this through topology-aware scheduling: when a StorageClass sets volumeBindingMode: WaitForFirstConsumer, the PV is not provisioned until the Pod is scheduled, ensuring the volume is created in the same AZ as the node.

For StatefulSets, this means each replica can land in a different AZ with its volume in the matching zone — exactly what you want for high availability. However, if a node in one AZ goes down, the Pod cannot be rescheduled to another AZ because its volume is stuck in the original zone. Plan for this by either using replicated storage (Ceph, Longhorn) that spans AZs, or by accepting that recovery requires restoring from a snapshot in the new AZ.

Testing Storage Failover

Do not wait for a real outage to find out your storage layer fails ungracefully. Test these scenarios regularly:

• Node drain: Run kubectl drain on a node with active PVs. Verify the volume detaches cleanly and the replacement Pod mounts it on the new node within your SLA (typical: 1–3 minutes for EBS).
• Force-delete a Pod: Use kubectl delete pod --force --grace-period=0. Confirm the volume attachment is released and the new Pod is not stuck in ContainerCreating waiting for the stale VolumeAttachment to expire (the default force-detach timeout is 6 minutes).
• AZ failure simulation: Cordon all nodes in one AZ. Verify that StatefulSet Pods with WaitForFirstConsumer volumes do not automatically recover (expected behavior), and that you can restore from a snapshot in another AZ.
• Snapshot restore: Restore a VolumeSnapshot into a new PVC and verify data integrity. If you cannot read back what you wrote, your backup strategy is broken.

Popular CSI Drivers

The CSI ecosystem is mature, with production-grade drivers available for all major cloud providers and several open-source storage systems. Here is a quick reference for the most widely deployed drivers:

When choosing a CSI driver, verify that it supports the specific features you need (snapshots, expansion, cloning, RWX) and check its Kubernetes version compatibility matrix. Run the driver's conformance tests in your CI pipeline before upgrading versions in production.

ConfigMaps and Secrets — Externalizing Configuration

The twelve-factor app methodology defines a strict boundary: configuration that varies between deployments must live outside your code. Database URLs, feature flags, API keys, TLS certificates — none of these belong in a container image. An image built once should run identically in dev, staging, and production. The only thing that changes is the configuration injected at runtime.

Kubernetes operationalizes this principle through two first-class objects: ConfigMaps for non-sensitive data and Secrets for sensitive data. Both decouple configuration from your container image, but they differ in how the cluster handles, stores, and exposes the data they carry.

ConfigMaps: Non-Sensitive Configuration

A ConfigMap is a namespaced Kubernetes object that stores key-value pairs. The values can be short strings (a feature flag, a log level) or entire files (an nginx.conf, a .properties file). The maximum size of a ConfigMap is 1 MiB.

Creating ConfigMaps

You can create ConfigMaps from multiple sources. Understanding the distinction matters because it determines how keys are named inside the resulting object.

From literal key-value pairs — useful for simple settings:

kubectl create configmap app-settings \
  --from-literal=LOG_LEVEL=info \
  --from-literal=MAX_RETRIES=3 \
  --from-literal=FEATURE_DARK_MODE=true

From a file — the filename becomes the key, the file content becomes the value:

# Key will be "nginx.conf", value will be the file contents
kubectl create configmap nginx-config --from-file=nginx.conf

# Override the key name explicitly
kubectl create configmap nginx-config --from-file=my-custom-key=nginx.conf

From a directory — each file in the directory becomes a key-value pair:

# Every file in ./config/ becomes a key
kubectl create configmap app-config --from-file=./config/

From an env file — parses KEY=VALUE lines (ignores comments and blank lines):

kubectl create configmap app-env --from-env-file=app.env

The declarative equivalent in YAML gives you version-controlled, reproducible configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-settings
  namespace: production
data:
  LOG_LEVEL: "info"
  MAX_RETRIES: "3"
  nginx.conf: |
    server {
        listen 80;
        server_name example.com;
        location / {
            proxy_pass http://backend:8080;
        }
    }

Consuming ConfigMaps as Environment Variables

There are two approaches to injecting ConfigMap data as environment variables, and choosing between them affects maintainability and naming control.

envFrom injects every key in the ConfigMap as an environment variable. It is convenient but gives you less control — if someone adds a key to the ConfigMap, it automatically appears in your container's environment.

apiVersion: v1
kind: Pod
metadata:
  name: web-app
spec:
  containers:
    - name: app
      image: myapp:1.4.0
      envFrom:
        - configMapRef:
            name: app-settings
          prefix: CFG_  # optional: prepends CFG_ to every key

Individual env entries with valueFrom give you explicit control. You pick exactly which keys to expose and can rename them in the process.

spec:
  containers:
    - name: app
      image: myapp:1.4.0
      env:
        - name: APP_LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-settings
              key: LOG_LEVEL
              optional: true  # pod starts even if key is missing

Consuming ConfigMaps as Volume Mounts

When your application reads configuration from files (not environment variables), mount the ConfigMap as a volume. Each key in the ConfigMap becomes a file in the mount directory, and the value becomes the file's content.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:1.25
      volumeMounts:
        - name: config-volume
          mountPath: /etc/nginx/conf.d
          readOnly: true
  volumes:
    - name: config-volume
      configMap:
        name: nginx-config
        items:                  # optional: select specific keys
          - key: nginx.conf
            path: default.conf  # rename inside the mount

The items field lets you project specific keys and rename them in the mount. Without items, every key in the ConfigMap appears as a file. You can also use subPath to mount a single file without overwriting the entire target directory — but be aware that subPath mounts do not receive automatic updates.

ConfigMaps as Command-Line Arguments

You can reference ConfigMap values in a container's command or args by first mapping them to environment variables and then using the $(VAR_NAME) substitution syntax.

spec:
  containers:
    - name: worker
      image: myworker:2.1.0
      env:
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-settings
              key: LOG_LEVEL
      command: ["./worker"]
      args: ["--log-level", "$(LOG_LEVEL)", "--max-retries", "5"]

Secrets: Sensitive Configuration

Secrets look almost identical to ConfigMaps in terms of consumption (env vars, volumes, command args). The critical difference is intent: Secrets signal to the cluster that the data is sensitive. Kubernetes responds by storing them in tmpfs on nodes (never written to disk on the node), restricting access through RBAC, and — if configured — encrypting them at rest in etcd.

Secret Types

Kubernetes defines several built-in Secret types. The type constrains what keys the Secret must contain and enables specialized behavior in controllers and the kubelet.

Creating Secrets

The imperative approach mirrors ConfigMap creation. Kubernetes automatically base64-encodes the values when you use kubectl create secret.

# Generic (Opaque) secret
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password='S3cur3P@ss!'

# TLS secret from certificate files
kubectl create secret tls app-tls \
  --cert=tls.crt \
  --key=tls.key

# Docker registry secret
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=deploy \
  --docker-password=token123

In declarative YAML, you must base64-encode values yourself in the data field — or use stringData for plain text that Kubernetes encodes on submission:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
# data: values must be base64-encoded
data:
  username: YWRtaW4=           # echo -n 'admin' | base64
  password: UzNjdXIzUEBzcyE=  # echo -n 'S3cur3P@ss!' | base64
---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials-easy
type: Opaque
# stringData: plain text — Kubernetes encodes it for you
stringData:
  username: admin
  password: "S3cur3P@ss!"

Consuming Secrets

Secrets are consumed identically to ConfigMaps — through envFrom, individual env entries, and volume mounts. The only difference in the YAML is that you reference secretKeyRef instead of configMapKeyRef, or use a secret volume source.

apiVersion: v1
kind: Pod
metadata:
  name: api-server
spec:
  containers:
    - name: api
      image: myapi:3.0.0
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
      volumeMounts:
        - name: tls-certs
          mountPath: /etc/tls
          readOnly: true
  volumes:
    - name: tls-certs
      secret:
        secretName: app-tls
        defaultMode: 0400  # restrict file permissions

Setting defaultMode: 0400 ensures that only the container's user can read the mounted Secret files. This is a simple but effective defense-in-depth measure. For registry credentials, reference the Secret in the pod's imagePullSecrets field rather than mounting it.

Encryption at Rest

By default, Secrets are stored unencrypted in etcd. Anyone with direct etcd access can read every Secret in the cluster. To fix this, you configure an EncryptionConfiguration on the API server that encrypts Secret data before it is written to etcd.

# /etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}  # fallback: read unencrypted data

The API server loads this file via the --encryption-provider-config flag. The providers list is ordered: the first provider is used for writing, and all providers are tried for reading (which is how you rotate keys gracefully). Common providers include aescbc, aesgcm, secretbox, and kms (for integrating with cloud KMS services like AWS KMS, GCP Cloud KMS, or Azure Key Vault).

After enabling encryption, existing Secrets remain unencrypted until you re-write them. Force re-encryption of all Secrets with:

kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

External Secret Management

For production clusters, encryption at rest is the minimum. Most teams go further by keeping secret values entirely outside of Kubernetes and syncing them in at runtime. This avoids having sensitive values in etcd at all, and centralizes secret lifecycle management (rotation, auditing, access control) in a dedicated vault system.

Here is a minimal External Secrets Operator example syncing a database password from AWS Secrets Manager:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials          # resulting K8s Secret name
    creationPolicy: Owner
  data:
    - secretKey: username         # key in the K8s Secret
      remoteRef:
        key: prod/db-credentials  # path in AWS Secrets Manager
        property: username
    - secretKey: password
      remoteRef:
        key: prod/db-credentials
        property: password

Immutable ConfigMaps and Secrets

Kubernetes 1.21 graduated the immutable field to stable. Setting immutable: true on a ConfigMap or Secret provides two concrete benefits:

• Performance — The kubelet stops polling the API server for updates to that object. In clusters with thousands of ConfigMaps, this significantly reduces API server load and watch traffic.
• Safety — Accidental or malicious edits are blocked. The only way to change the configuration is to create a new object with a new name and update the pods referencing it.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-settings-v3
immutable: true
data:
  LOG_LEVEL: "warn"
  MAX_RETRIES: "5"
---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials-v2
immutable: true
type: Opaque
stringData:
  username: admin
  password: "NewS3cur3P@ss!"

The common pattern is to append a version suffix or content hash to the name (app-settings-v3, app-settings-a8f3d). Tools like Kustomize automate this with configMapGenerator and the --append-hash behavior, producing names like app-settings-k5m8h and updating all Deployment references automatically. This triggers a rolling update whenever config changes — giving you the equivalent of a config-driven redeployment.

Putting It All Together

Here is a realistic Deployment that uses a ConfigMap for application settings, a Secret for database credentials, and a volume-mounted ConfigMap for a custom configuration file — all in one spec.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
        - name: app
          image: order-service:4.2.1
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: order-settings   # LOG_LEVEL, MAX_RETRIES, etc.
          env:
            - name: DB_HOST
              valueFrom:
                configMapKeyRef:
                  name: order-settings
                  key: DB_HOST
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: order-db-creds
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: order-db-creds
                  key: password
          volumeMounts:
            - name: app-config
              mountPath: /etc/order-service
              readOnly: true
            - name: tls-certs
              mountPath: /etc/tls
              readOnly: true
      volumes:
        - name: app-config
          configMap:
            name: order-service-config
        - name: tls-certs
          secret:
            secretName: order-tls
            defaultMode: 0400

Resource Requests, Limits, and Quality of Service

Every container in a Kubernetes cluster competes for finite CPU and memory on the nodes it runs on. Without explicit resource declarations, the scheduler is flying blind — it cannot make intelligent placement decisions, and the kubelet cannot protect one workload from another's greed. Resource requests and limits are how you communicate your workload's needs to Kubernetes, and they directly determine scheduling, runtime enforcement, and eviction priority.

Requests vs. Limits — Two Different Guarantees

A request is the amount of CPU or memory that Kubernetes guarantees to a container. The scheduler uses requests to decide which node has enough room to place a pod. If a node has 4 CPU cores and existing pods have requested 3.5 cores total, the scheduler will only place a new pod there if its CPU request is 0.5 cores or less.

A limit is the maximum amount a container is allowed to consume at runtime. The kubelet enforces limits through kernel mechanisms — Linux cgroups. A container can use resources up to its limit, but never beyond. The request is a floor for scheduling; the limit is a ceiling for enforcement.

Here is a pod spec that sets both requests and limits for CPU and memory:

apiVersion: v1
kind: Pod
metadata:
  name: web-server
spec:
  containers:
    - name: nginx
      image: nginx:1.27
      resources:
        requests:
          cpu: "250m"       # 0.25 CPU cores guaranteed
          memory: "128Mi"   # 128 MiB guaranteed
        limits:
          cpu: "500m"       # Can burst up to 0.5 cores
          memory: "256Mi"   # Hard cap — OOMKilled if exceeded

CPU Resources: Millicores and CFS Throttling

CPU is measured in millicores (or millicpu). One core equals 1000m. You can express CPU as a decimal (0.5) or in millicores (500m) — they are equivalent. Unlike memory, CPU is a compressible resource: when a container hits its CPU limit, it is not killed — it is throttled.

Throttling is enforced through the Linux kernel's Completely Fair Scheduler (CFS) bandwidth control. The CFS works in periods (typically 100ms). If a container has a CPU limit of 500m, it receives a quota of 50ms per 100ms period. Once the container exhausts its quota within a period, the kernel suspends its threads until the next period begins. The container stays alive but gets slower — its processes stall mid-execution waiting for their next CPU slice.

Memory Resources: Bytes and OOMKill

Memory is measured in bytes, using standard suffixes: Ki, Mi, Gi (power-of-two) or K, M, G (power-of-ten). Always use the binary suffixes (Mi, Gi) to avoid confusion — 128Mi is 134,217,728 bytes, while 128M is 128,000,000 bytes. Memory is incompressible: unlike CPU, you cannot simply slow a process down when it uses too much memory. The kernel must reclaim the memory, and it does so violently.

When a container's memory usage exceeds its limit, the Linux kernel's OOM (Out of Memory) killer terminates the container's process. Kubernetes sees this as a container failure with exit code 137 (128 + SIGKILL signal 9) and reason OOMKilled. If the pod's restartPolicy allows it, the kubelet restarts the container — but if it keeps getting OOMKilled, Kubernetes applies exponential back-off (CrashLoopBackOff).

Quality of Service Classes

Kubernetes automatically assigns every pod a QoS class based on the resource requests and limits of its containers. The QoS class determines eviction priority when a node is under memory pressure — Kubernetes kills lower-priority pods first to free resources for higher-priority ones. You never set the QoS class directly; it is computed from your resource configuration.

The following diagram shows how Kubernetes determines the QoS class:

flowchart TD
    A["Pod Created"] --> B{"All containers have
CPU & memory requests
AND limits set?"}
    B -- No --> C{"Any container has at
least one request
or limit set?"}
    B -- Yes --> D{"requests == limits
for every container?"}
    C -- No --> E["BestEffort
Lowest priority — evicted first"]
    C -- Yes --> F["Burstable
Medium priority"]
    D -- Yes --> G["Guaranteed
Highest priority — evicted last"]
    D -- No --> F
    

Guaranteed QoS Example

For Guaranteed QoS, every container must declare both CPU and memory with requests exactly equal to limits. This is the most predictable configuration — the container gets a fixed, reserved allocation.

apiVersion: v1
kind: Pod
metadata:
  name: payment-processor
spec:
  containers:
    - name: processor
      image: payments:3.2.1
      resources:
        requests:
          cpu: "1"
          memory: "512Mi"
        limits:
          cpu: "1"          # Same as request → Guaranteed
          memory: "512Mi"   # Same as request → Guaranteed

Burstable QoS Example

Most production workloads fall into Burstable. You set requests to size for typical load and limits to allow headroom for spikes.

apiVersion: v1
kind: Pod
metadata:
  name: api-server
spec:
  containers:
    - name: api
      image: myapp-api:2.1.0
      resources:
        requests:
          cpu: "250m"       # Typical steady-state usage
          memory: "256Mi"
        limits:
          cpu: "1"          # Allow 4x burst for traffic spikes
          memory: "512Mi"   # Double the request for safety

BestEffort QoS Example

A pod with no resource declarations at all gets BestEffort. It can use whatever is available but will be the first to be evicted under memory pressure.

apiVersion: v1
kind: Pod
metadata:
  name: batch-worker
spec:
  containers:
    - name: worker
      image: data-cruncher:latest
      # No resources block at all → BestEffort

You can verify a pod's QoS class at any time:

kubectl get pod payment-processor -o jsonpath='{.status.qosClass}'
# Output: Guaranteed

LimitRanges — Default and Boundary Guardrails

Relying on every developer to remember setting requests and limits is fragile. A LimitRange is a namespace-scoped policy that defines default values, minimum values, and maximum values for resource requests and limits on containers and pods. When a pod is created without explicit resource declarations, the LimitRange admission controller injects the defaults automatically.

apiVersion: v1
kind: LimitRange
metadata:
  name: container-limits
  namespace: production
spec:
  limits:
    - type: Container
      default:           # Applied as limits if not specified
        cpu: "500m"
        memory: "256Mi"
      defaultRequest:    # Applied as requests if not specified
        cpu: "100m"
        memory: "128Mi"
      min:               # Reject pods requesting less than this
        cpu: "50m"
        memory: "64Mi"
      max:               # Reject pods requesting more than this
        cpu: "4"
        memory: "4Gi"
    - type: Pod
      max:               # Total across all containers in a pod
        cpu: "8"
        memory: "8Gi"

If a developer deploys a container with no resource configuration into the production namespace, the LimitRange controller automatically sets requests.cpu: 100m, requests.memory: 128Mi, limits.cpu: 500m, and limits.memory: 256Mi. If they try to request cpu: 10 (10 cores), the API server rejects the pod with a validation error because it exceeds the max constraint.

ResourceQuotas — Namespace-Level Budgets

While LimitRanges constrain individual containers and pods, ResourceQuotas constrain an entire namespace's aggregate consumption. They prevent a single team or application from monopolizing cluster resources. A ResourceQuota specifies the total amount of compute resources, storage, and even object counts that all pods in a namespace can collectively consume.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
  namespace: team-alpha
spec:
  hard:
    # Compute totals
    requests.cpu: "20"         # Max 20 CPU cores of requests
    requests.memory: "40Gi"    # Max 40 GiB of memory requests
    limits.cpu: "40"           # Max 40 CPU cores of limits
    limits.memory: "80Gi"      # Max 80 GiB of memory limits

    # Object counts
    pods: "50"                 # Max 50 pods in namespace
    services: "20"             # Max 20 services
    configmaps: "30"           # Max 30 ConfigMaps
    persistentvolumeclaims: "10"
    secrets: "30"

    # Storage
    requests.storage: "200Gi"  # Total PVC storage

Check current usage against the quota:

kubectl describe resourcequota team-alpha-quota -n team-alpha

# Name:                  team-alpha-quota
# Resource               Used    Hard
# --------               ----    ----
# limits.cpu             12      40
# limits.memory          28Gi    80Gi
# pods                   18      50
# requests.cpu           6       20
# requests.memory        14Gi    40Gi

Ephemeral Storage Requests and Limits

Beyond CPU and memory, containers consume local disk space for logs, temporary files, and writable layers. Kubernetes tracks this as ephemeral-storage — space consumed on the node's root filesystem. You can set requests and limits for ephemeral storage the same way you do for CPU and memory. If a container exceeds its ephemeral-storage limit, the kubelet evicts the pod.

apiVersion: v1
kind: Pod
metadata:
  name: log-processor
spec:
  containers:
    - name: processor
      image: log-processor:1.4.0
      resources:
        requests:
          cpu: "500m"
          memory: "256Mi"
          ephemeral-storage: "1Gi"   # Need at least 1 GiB disk
        limits:
          cpu: "1"
          memory: "512Mi"
          ephemeral-storage: "2Gi"   # Evicted if exceeds 2 GiB

Ephemeral-storage accounting includes the container's writable layer, log files (/var/log), and emptyDir volumes (unless they are backed by tmpfs or a dedicated medium). This is particularly important for workloads that write large temporary files — image processing pipelines, build agents, or log-heavy applications — where unchecked disk usage could fill the node's filesystem and impact all pods on that node.

The CPU Limits Debate

There is an active and legitimate debate in the Kubernetes community about whether you should set CPU limits at all. Both sides have valid arguments, and the right answer depends on your workload characteristics and priorities.

The Case Against CPU Limits

The argument: if a node has idle CPU capacity, why prevent a container from using it? CPU limits enforce a hard ceiling via CFS quota, meaning a container gets throttled even when no other workload wants those CPU cycles. This leads to wasted capacity and unnecessary latency. Companies like Google (in Borg, the predecessor to Kubernetes) and several prominent community voices recommend setting only CPU requests and omitting CPU limits entirely.

The Case For CPU Limits

The counter-argument: without limits, a single misbehaving container (an infinite loop, a regex backtrack, a memory leak in a JIT compiler) can consume all available CPU on a node, degrading every other pod's performance. CPU requests only guarantee a minimum share — they do not prevent a container from consuming far more during contention. Limits provide predictability and isolation, which matters when you have mixed workloads from different teams.

Putting It All Together

In practice, you rarely configure resources in isolation. A well-managed namespace uses all three primitives together: LimitRange for sane defaults, ResourceQuota for budget enforcement, and explicit resource declarations in your pod specs for precision. Here is a complete namespace setup:

apiVersion: v1
kind: Namespace
metadata:
  name: team-backend
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-backend
spec:
  limits:
    - type: Container
      default:
        cpu: "500m"
        memory: "256Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      max:
        cpu: "4"
        memory: "4Gi"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: team-backend
spec:
  hard:
    requests.cpu: "16"
    requests.memory: "32Gi"
    limits.cpu: "32"
    limits.memory: "64Gi"
    pods: "40"

With this configuration, any pod deployed to team-backend without resource declarations automatically gets requests of 100m CPU / 128Mi memory and limits of 500m CPU / 256Mi memory. No single container can exceed 4 cores or 4Gi memory. The namespace as a whole cannot exceed 16 cores of CPU requests or 40 pods total. This layered approach gives you safety by default while letting individual workloads override values within the defined guardrails.

Namespaces, Labels, and Annotations — Organizing Resources

A running Kubernetes cluster quickly accumulates hundreds — sometimes thousands — of resources. Without organizational structure, finding the right Deployment or debugging a failing Service becomes a needle-in-a-haystack problem. Kubernetes provides three complementary mechanisms to keep things manageable: Namespaces partition the cluster logically, Labels tag resources for selection and grouping, and Annotations attach arbitrary metadata for tools and humans.

These three features operate at different levels. Namespaces are a hard boundary enforced by the API server — they affect resource visibility, access control, and resource quotas. Labels are soft, queryable tags that controllers and kubectl use to match resources together. Annotations are freeform metadata that have no effect on selection or scheduling but carry essential information for external tools, operators, and auditing.

Namespaces: Logical Cluster Partitions

A namespace is a virtual cluster inside your physical cluster. Resources in one namespace are invisible to kubectl commands scoped to another namespace (unless you explicitly ask). This isolation makes namespaces the primary tool for multi-tenancy, environment separation, and organizational boundaries.

The Four Built-in Namespaces

Every Kubernetes cluster ships with four namespaces out of the box, each serving a specific purpose:

When to Create New Namespaces

The right namespace strategy depends on your organization. There is no single correct answer, but three patterns cover most real-world cases:

• Per-team — team-backend, team-frontend, team-data. Good when teams own distinct services and you want to apply separate RBAC policies and resource quotas per team.
• Per-environment — dev, staging, production. Simple, but only works well for small clusters. In larger organizations, environments typically live in separate clusters entirely.
• Per-application — checkout-service, payment-service. Provides fine-grained isolation and makes it easy to tear down everything related to one application at once.

Creating a namespace is straightforward:

# Imperative
kubectl create namespace team-backend

# Declarative (preferred for GitOps)
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: team-backend
  labels:
    team: backend
    environment: production
EOF

Namespace-Scoped vs. Cluster-Scoped Resources

Not every Kubernetes resource lives inside a namespace. Resources that apply to the entire cluster — like Nodes, PersistentVolumes, ClusterRoles, and Namespaces themselves — are cluster-scoped. Most workload resources (Pods, Deployments, Services, ConfigMaps, Secrets) are namespace-scoped.

# List all namespace-scoped resource types
kubectl api-resources --namespaced=true

# List all cluster-scoped resource types
kubectl api-resources --namespaced=false

Cross-Namespace Communication

Namespaces are a logical boundary, not a network boundary. By default, a Pod in namespace team-backend can freely communicate with a Service in namespace team-frontend. The key is DNS: Kubernetes DNS gives every Service a fully qualified name following the pattern <service>.<namespace>.svc.cluster.local.

# Within the same namespace, short names work
curl http://payment-api:8080/health

# From a different namespace, use the FQDN
curl http://payment-api.team-backend.svc.cluster.local:8080/health

Labels: Identifying and Selecting Resources

Labels are key-value pairs attached to any Kubernetes object's metadata. Unlike namespaces, which provide hard partitions, labels are a flexible tagging system. Their real power comes from selectors — the mechanism that controllers, Services, and kubectl use to find matching resources.

Every major Kubernetes abstraction depends on labels. A Service routes traffic to Pods that match its selector. A Deployment manages ReplicaSets through label matching. A DaemonSet picks which nodes to run on using node labels. Understanding labels means understanding how Kubernetes wires everything together.

Label Syntax Rules

Labels follow strict formatting requirements. Keys can have an optional prefix (a DNS subdomain up to 253 characters) separated by a / from the name. The name portion must be 63 characters or fewer and match the pattern [a-z0-9A-Z] with -, _, and . allowed in the middle. Values follow the same rules but can also be empty.

metadata:
  labels:
    app: payment-api           # simple key
    version: v2.3.1            # version tracking
    tier: backend              # architectural layer
    app.kubernetes.io/name: payment-api        # recommended label (prefixed)
    app.kubernetes.io/component: server
    app.kubernetes.io/managed-by: helm

Recommended Labels

Kubernetes defines a set of recommended labels under the app.kubernetes.io prefix. These labels are not enforced by the system, but they create a shared vocabulary that tools like Helm, ArgoCD, and various dashboards understand.

Label Selectors

Selectors are how Kubernetes answers the question "which resources match?" There are two flavors: equality-based and set-based. Equality-based selectors use =, ==, and !=. Set-based selectors use in, notin, and exists. Both can be combined in a single query — all conditions must be satisfied (logical AND).

# Equality-based: find all pods for the payment-api app
kubectl get pods -l app=payment-api

# Inequality: everything except the frontend tier
kubectl get pods -l tier!=frontend

# Set-based: pods in either staging or production
kubectl get pods -l 'environment in (staging, production)'

# Set-based: pods that have a "release" label (any value)
kubectl get pods -l 'release'

# Set-based: pods without a "canary" label
kubectl get pods -l '!canary'

# Combining selectors (AND logic)
kubectl get pods -l 'app=payment-api,environment in (production)'

How Controllers Use Selectors

The selector mechanism is the glue that holds Kubernetes' declarative model together. A Deployment does not directly manage Pods — it manages ReplicaSets, which in turn manage Pods. The connection at each level is made through label selectors. Here is a concrete example showing the full chain:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  replicas: 3
  selector:
    matchLabels:              # Deployment finds ReplicaSets with these labels
      app: payment-api
  template:
    metadata:
      labels:                 # Pods created with these labels — must match selector above
        app: payment-api
        version: v2.3.1
    spec:
      containers:
        - name: api
          image: myregistry/payment-api:2.3.1
---
apiVersion: v1
kind: Service
metadata:
  name: payment-api
spec:
  selector:                   # Service routes to Pods matching these labels
    app: payment-api
  ports:
    - port: 80
      targetPort: 8080

The Deployment's spec.selector.matchLabels must be a subset of the Pod template's metadata.labels. If they don't match, the API server rejects the manifest. The Service's spec.selector independently matches Pods by label — it has no knowledge of the Deployment at all. This loose coupling is intentional: a Service can route to Pods managed by different Deployments, StatefulSets, or even bare Pods, as long as the labels match.

Annotations: Metadata for Tools and Humans

Annotations look like labels — they are key-value pairs in metadata — but they serve a fundamentally different purpose. You cannot select or filter resources by annotations. Instead, annotations carry non-identifying information that tools, controllers, and operators read at runtime. Think of them as a structured comment attached to a resource.

Annotations have relaxed constraints compared to labels. Values can be much larger (up to 256 KB total for all annotations on a resource) and can contain any UTF-8 characters, including JSON, URLs, and multi-line strings.

Common Annotations in the Wild

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  annotations:
    kubernetes.io/change-cause: "Upgraded to v2.3.1 — fixes CVE-2024-1234"
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: api
          image: myregistry/payment-api:2.3.1

Labels vs. Annotations: When to Use Which

The decision rule is simple: if Kubernetes or your tooling needs to select or group resources by this piece of metadata, use a label. If the data is informational, consumed by external tools, or too large for a label, use an annotation.

Practical: Managing Labels with kubectl

Beyond filtering with -l, kubectl gives you commands to add, update, and remove labels on live resources — useful for quick operational tasks like marking a Pod for debugging or excluding it from a Service's traffic.

# Add a label to a running Pod
kubectl label pod payment-api-7d6f8b4c5-x9kzq debug=true

# Overwrite an existing label (--overwrite is required)
kubectl label pod payment-api-7d6f8b4c5-x9kzq version=v2.4.0 --overwrite

# Remove a label (use the key followed by a minus sign)
kubectl label pod payment-api-7d6f8b4c5-x9kzq debug-

# Show labels as columns in output
kubectl get pods --show-labels

# Use a custom column to display a specific label
kubectl get pods -L app,version

# Label all pods in a namespace at once
kubectl label pods --all environment=staging -n team-backend

Similarly, you can manage annotations imperatively:

# Add an annotation
kubectl annotate deployment payment-api kubernetes.io/change-cause="Rollback to v2.2.0"

# Remove an annotation
kubectl annotate deployment payment-api kubernetes.io/change-cause-

# View annotations on a resource
kubectl get deployment payment-api -o jsonpath='{.metadata.annotations}'

Putting It All Together

In practice, you use all three mechanisms in concert. Namespaces divide ownership and access. Labels connect resources and enable querying. Annotations carry the metadata that your CI/CD pipeline, monitoring stack, and Ingress controllers rely on.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: team-backend                        # Namespace: ownership boundary
  labels:                                        # Labels: identification & selection
    app.kubernetes.io/name: payment-api
    app.kubernetes.io/version: "2.3.1"
    app.kubernetes.io/component: server
    app.kubernetes.io/part-of: e-commerce
    app.kubernetes.io/managed-by: argocd
  annotations:                                   # Annotations: tool metadata
    kubernetes.io/change-cause: "Release 2.3.1"
    argocd.argoproj.io/sync-wave: "2"
    git.commit/sha: "a1b2c3d4e5f6"
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: payment-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: payment-api
        app.kubernetes.io/version: "2.3.1"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
    spec:
      containers:
        - name: api
          image: myregistry/payment-api:2.3.1
          ports:
            - containerPort: 8080
            - containerPort: 9090
              name: metrics

With this foundation — namespaces for boundaries, labels for selection, and annotations for metadata — you have the organizational toolkit to keep a complex cluster understandable as it scales. The next section builds directly on namespaces by introducing RBAC and ServiceAccounts, which control who can access the resources inside each namespace.

RBAC and ServiceAccounts — Who Can Do What

Every request that hits the Kubernetes API server passes through three gates: authentication (who are you?), authorization (are you allowed to do this?), and admission control (should we modify or reject this request?). This section focuses on the first two gates — specifically, how Kubernetes identifies callers and how RBAC grants or denies their actions.

Getting RBAC wrong is one of the fastest paths to a cluster security incident. Overly permissive ClusterRoleBindings to cluster-admin are disturbingly common in production. Understanding the model — and knowing how to scope permissions tightly — is essential.

Authentication — Proving Who You Are

Kubernetes does not have a built-in user database. It does not store usernames and passwords. Instead, the API server delegates identity verification to external systems through pluggable authentication strategies. When a request arrives, the API server tries each configured authenticator in order until one succeeds.

Authorization — Deciding What You Can Do

Once the API server knows who you are, it determines what you can do. Kubernetes supports multiple authorization modes, configured via the --authorization-mode flag on the API server. The modes are evaluated in order, and the first one that makes a decision (allow or deny) wins.

In practice, you will work with RBAC on every cluster. The rest of this section is a deep dive into the RBAC model.

The RBAC Model

RBAC in Kubernetes is built from four resource types that connect in a clear pattern: Roles define what actions are allowed, and Bindings connect those roles to subjects (users, groups, or ServiceAccounts). The "namespace vs. cluster" axis gives you two levels of scope.

erDiagram
    Role {
        string namespace
        string name
        list rules
    }
    ClusterRole {
        string name
        list rules
    }
    RoleBinding {
        string namespace
        string roleRef
        list subjects
    }
    ClusterRoleBinding {
        string roleRef
        list subjects
    }
    User {
        string name
    }
    Group {
        string name
    }
    ServiceAccount {
        string namespace
        string name
    }

    Role ||--o{ RoleBinding : "referenced by"
    ClusterRole ||--o{ RoleBinding : "referenced by"
    ClusterRole ||--o{ ClusterRoleBinding : "referenced by"
    RoleBinding }o--|| User : "grants to"
    RoleBinding }o--|| Group : "grants to"
    RoleBinding }o--|| ServiceAccount : "grants to"
    ClusterRoleBinding }o--|| User : "grants to"
    ClusterRoleBinding }o--|| Group : "grants to"
    ClusterRoleBinding }o--|| ServiceAccount : "grants to"
    

There are four key resources to understand:

A subtle but powerful pattern: a RoleBinding can reference a ClusterRole. This lets you define a reusable set of permissions once as a ClusterRole and then grant it in specific namespaces through RoleBindings. The subject only gets those permissions within the RoleBinding's namespace — not cluster-wide.

RBAC Verbs

Each rule in a Role or ClusterRole specifies which verbs (actions) are allowed on which resources in which API groups. Kubernetes defines eight verbs that map directly to HTTP methods on the API server.

Common groupings: read-only access typically means get, list, watch. Full management adds create, update, patch, delete. The wildcard * matches all verbs — use it sparingly.

Role and ClusterRole

A Role grants permissions within a specific namespace. Each rule in the rules array specifies API groups, resources, and the verbs allowed on those resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: staging
  name: deployment-manager
rules:
  # Full control over Deployments
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  # Read-only access to Pods and their logs
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]

The apiGroups field determines which API group the resources belong to. Core resources (Pods, Services, ConfigMaps) use "" (the empty string). Resources in named groups use the group name — "apps" for Deployments, "batch" for Jobs, "networking.k8s.io" for NetworkPolicies.

A ClusterRole looks identical but has no namespace field and can also grant access to cluster-scoped resources like Nodes, PersistentVolumes, and Namespaces themselves.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]

You can also restrict access to specific resource names using the resourceNames field. This is useful for granting access to a particular ConfigMap or Secret without opening up all resources of that type.

rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["app-config", "feature-flags"]
    verbs: ["get", "update"]

RoleBinding and ClusterRoleBinding

Roles and ClusterRoles are inert until you bind them to subjects. A RoleBinding grants the permissions defined in a Role (or ClusterRole) to one or more subjects — within a single namespace. A ClusterRoleBinding grants permissions cluster-wide.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deploy-manager-binding
  namespace: staging
subjects:
  # A specific user (from certificate CN or OIDC)
  - kind: User
    name: jane.doe
    apiGroup: rbac.authorization.k8s.io
  # A group (from certificate O field or OIDC groups claim)
  - kind: Group
    name: platform-team
    apiGroup: rbac.authorization.k8s.io
  # A ServiceAccount in the same namespace
  - kind: ServiceAccount
    name: ci-deployer
    namespace: staging
roleRef:
  kind: Role
  name: deployment-manager
  apiGroup: rbac.authorization.k8s.io

The roleRef is immutable — once a Binding is created, you cannot change which Role it references. To point it at a different Role, delete and recreate the Binding. This prevents privilege escalation through in-place modification.

Here is a ClusterRoleBinding that grants node-reader to the monitoring group across the entire cluster:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: monitoring-node-reader
subjects:
  - kind: Group
    name: monitoring
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

ServiceAccounts

ServiceAccounts are the identity mechanism for workloads running inside the cluster. Every namespace has a default ServiceAccount, and every Pod runs as some ServiceAccount. When a Pod makes API calls (e.g., a controller querying Pods, or a CI tool creating Deployments), it authenticates using its ServiceAccount token.

Automatic Token Mounting

By default, Kubernetes mounts a ServiceAccount token into every Pod at /var/run/secrets/kubernetes.io/serviceaccount/. Since Kubernetes 1.22, these are bound service account tokens — time-limited JWTs projected through the TokenRequest API, scoped to a specific audience and expiration (default: 1 hour, auto-refreshed by the kubelet).

# Inspect the projected token volume inside a running Pod
kubectl exec my-pod -- ls /var/run/secrets/kubernetes.io/serviceaccount/
# Output: ca.crt  namespace  token

# Decode the JWT to see its claims (bound, time-limited)
kubectl exec my-pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/token \
  | cut -d'.' -f2 | base64 -d 2>/dev/null | python3 -m json.tool

Disabling Automatic Token Mounting

Most application Pods never call the Kubernetes API. Mounting a token into these Pods is an unnecessary attack surface — if the Pod is compromised, the attacker gets a valid API token. Disable it at the ServiceAccount level, the Pod level, or both.

# Option 1: Disable at the ServiceAccount level (affects all Pods using it)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: production
automountServiceAccountToken: false
---
# Option 2: Disable at the Pod spec level (overrides ServiceAccount setting)
apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  serviceAccountName: my-app
  automountServiceAccountToken: false
  containers:
    - name: app
      image: my-app:1.4.0

Bound Service Account Token Volume Projection

Modern Kubernetes clusters (1.22+) use projected volumes to mount ServiceAccount tokens. Unlike the legacy approach of long-lived Secret-based tokens, projected tokens are issued on-demand through the TokenRequest API with three important properties:

• Time-limited — tokens expire (default 1 hour) and are automatically rotated by the kubelet before expiration.
• Audience-bound — tokens are scoped to a specific audience (typically the API server), preventing misuse with other services.
• Object-bound — tokens are tied to the specific Pod; if the Pod is deleted, the token becomes invalid immediately.

You can also request tokens with custom audiences and expiration times using a projected volume explicitly:

apiVersion: v1
kind: Pod
metadata:
  name: vault-consumer
spec:
  serviceAccountName: vault-auth
  containers:
    - name: app
      image: my-app:2.0.0
      volumeMounts:
        - name: vault-token
          mountPath: /var/run/secrets/vault
          readOnly: true
  volumes:
    - name: vault-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600
              audience: vault

Aggregated ClusterRoles

Aggregated ClusterRoles let you compose a ClusterRole from multiple smaller ClusterRoles using label selectors. Instead of maintaining one monolithic role, you define granular roles and aggregate them automatically. Kubernetes' built-in admin, edit, and view ClusterRoles use this mechanism — which is why installing a CRD can automatically make its resources visible in those roles.

# Parent: aggregates all ClusterRoles with the matching label
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-aggregate
aggregationRule:
  clusterRoleSelectors:
    - matchLabels:
        rbac.example.com/aggregate-to-monitoring: "true"
rules: []  # Rules are auto-populated by the controller
---
# Child: contributes rules to the aggregate
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-metrics-reader
  labels:
    rbac.example.com/aggregate-to-monitoring: "true"
rules:
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]
---
# Another child: automatically merged into the aggregate
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-events-reader
  labels:
    rbac.example.com/aggregate-to-monitoring: "true"
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch"]

The parent ClusterRole's rules field is automatically filled by the RBAC controller. Leave it as an empty array — any manually added rules will be overwritten. Adding or removing a child ClusterRole with the matching label dynamically updates the aggregate.

Auditing Permissions with kubectl auth

You do not need to read through dozens of Roles and Bindings to understand what a subject can do. The kubectl auth can-i command answers permission questions directly.

# Can I create deployments in the staging namespace?
kubectl auth can-i create deployments --namespace staging
# yes

# Can the "ci-deployer" ServiceAccount list pods in staging?
kubectl auth can-i list pods \
  --namespace staging \
  --as system:serviceaccount:staging:ci-deployer
# yes

# List ALL permissions for a specific ServiceAccount
kubectl auth can-i --list \
  --namespace staging \
  --as system:serviceaccount:staging:ci-deployer

# Check cluster-scoped permissions: can user jane delete nodes?
kubectl auth can-i delete nodes --as jane.doe
# no

# Check who you are currently authenticated as (Kubernetes 1.27+)
kubectl auth whoami

The --as flag triggers impersonation, which lets cluster admins test permissions from another identity's perspective. ServiceAccounts use the format system:serviceaccount:<namespace>:<name>. The --list flag outputs a table of all allowed actions — invaluable for auditing.

Practical Example: CI/CD Pipeline ServiceAccount

Here is a complete, production-ready example: a ServiceAccount for a CI/CD pipeline (like ArgoCD or a GitHub Actions runner) that can manage Deployments and Services in the staging namespace — and nothing else.

# 1. Create a dedicated ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: staging
---
# 2. Define the minimum required permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-deploy-role
  namespace: staging
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments/status"]
    verbs: ["get"]
---
# 3. Bind the Role to the ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deploy-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: staging
roleRef:
  kind: Role
  name: ci-deploy-role
  apiGroup: rbac.authorization.k8s.io

After applying this, verify the permissions are correct:

# Apply the manifests
kubectl apply -f ci-deployer-rbac.yaml

# Verify: can it create deployments in staging?
kubectl auth can-i create deployments \
  --namespace staging \
  --as system:serviceaccount:staging:ci-deployer
# yes

# Verify: can it delete secrets? (should be denied)
kubectl auth can-i delete secrets \
  --namespace staging \
  --as system:serviceaccount:staging:ci-deployer
# no

# Verify: can it create deployments in production? (should be denied)
kubectl auth can-i create deployments \
  --namespace production \
  --as system:serviceaccount:staging:ci-deployer
# no

# List all granted permissions
kubectl auth can-i --list \
  --namespace staging \
  --as system:serviceaccount:staging:ci-deployer

Built-in ClusterRoles Reference

Kubernetes ships with several default ClusterRoles. Understanding these helps you decide when to use a built-in role versus creating your own.

Pod Security Standards and Pod Security Admission

For years, PodSecurityPolicy (PSP) was the built-in mechanism for controlling what pods could and couldn't do on a Kubernetes cluster. It was powerful but deeply flawed — difficult to reason about, impossible to dry-run, and full of surprising interactions with RBAC. In Kubernetes 1.21, PSP was officially deprecated. In Kubernetes 1.25, it was removed entirely.

Its replacement is a two-part system: Pod Security Standards (PSS), which define three tiered security profiles, and Pod Security Admission (PSA), a built-in admission controller that enforces those standards at the namespace level. Together, they give you a simpler, more predictable way to prevent dangerous pod configurations from running on your cluster.

Why PodSecurityPolicy Had to Go

PSP suffered from fundamental design problems that made it unreliable in practice. Understanding why it failed helps you appreciate what PSA does differently.

• Confusing policy selection. When multiple PSPs existed, Kubernetes would silently pick one based on alphabetical ordering and mutation rules. Administrators couldn't easily predict which policy applied to a given pod.
• Tight RBAC coupling. PSPs were authorized through RBAC bindings on the use verb. This meant the effective policy depended on the identity creating the pod — not on the namespace or workload. A Deployment created via a controller used the controller's ServiceAccount, not the user's, leading to subtle bypasses.
• No dry-run or audit mode. You couldn't test a policy before enforcing it. Rolling out a new PSP to an existing cluster was a high-risk operation with no way to preview what would break.
• Mutation side effects. PSPs could mutate pod specs (setting default seccomp profiles, dropping capabilities), which made debugging unpredictable and conflicted with GitOps workflows expecting immutable manifests.

The Three Pod Security Standards

Pod Security Standards define three progressively restrictive security profiles. Each level is a superset of the one before it — Restricted includes everything Baseline checks, and Baseline includes everything Privileged allows (which is nothing, since Privileged is unrestricted). Think of them as presets, not custom policies.

Privileged — The Escape Hatch

The Privileged level applies zero restrictions. Any pod configuration is allowed: privileged containers, host namespaces, host paths — everything. This exists because certain infrastructure components (CNI plugins like Calico, storage provisioners like Longhorn, monitoring agents like the node exporter) genuinely need elevated access to the host.

Apply this level only to dedicated infrastructure namespaces like kube-system, and keep those namespaces tightly controlled via RBAC. Never use Privileged as the default for application namespaces.

Baseline — Sensible Defaults

Baseline blocks the most dangerous pod configurations while remaining compatible with the vast majority of application workloads. It prevents things like privileged containers, host networking, and dangerous capability additions, but still allows running as root and most volume types. Most off-the-shelf Helm charts and container images will pass Baseline without modification.

Restricted — Hardened Workloads

Restricted builds on Baseline and adds requirements that many existing images don't meet out of the box: pods must run as non-root, must drop ALL capabilities, must set a seccomp profile, and can only use a limited set of volume types. This is the level you want for multi-tenant clusters or environments with compliance requirements, but expect to update your Dockerfiles and pod specs to conform.

What Each Level Actually Checks

The specific checks at each level are defined in the Kubernetes documentation and are version-pinned. Here is a breakdown of the key controls and which level enforces them.