Kubernetes is the de facto platform for running containers in production. You describe the desired state of an application - which containers, how many copies, how they network, what resources they need - and Kubernetes continuously works to make reality match that description: scheduling containers onto machines, restarting failed ones, scaling under load, and replacing nodes that die. It came out of Google's internal cluster manager and is now governed by the CNCF.

The control loop is the whole idea

Everything in Kubernetes is a declarative object stored in etcd and reconciled by controllers. You submit "I want 5 replicas of this service"; a controller notices the current count is 3 and creates 2 more; if one crashes, the loop notices 4 and creates another. You almost never issue imperative "start this container here" commands - you edit the desired state and let the reconciler converge. This is what gives Kubernetes its self-healing behavior, and it is why config lives in version-controlled YAML rather than in someone's shell history.

The objects you actually use

  • Pod - the smallest unit: one or more tightly-coupled containers sharing a network namespace and storage. You rarely create pods directly.
  • Deployment - manages a replicated, self-healing set of identical pods and handles rolling updates and rollbacks. The workhorse for stateless services.
  • Service - a stable virtual IP and DNS name in front of a changing set of pods, so callers don't track individual pod IPs.
  • StatefulSet - for workloads needing stable identity and storage (databases, Kafka), giving each pod a sticky name and volume.
  • ConfigMap / Secret - inject configuration and credentials without baking them into images.
  • Job / CronJob - run-to-completion and scheduled batch tasks.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels: { app: api }
  template:
    metadata:
      labels: { app: api }
    spec:
      containers:
        - name: api
          image: registry/api:1.8.2
          resources:
            requests: { cpu: "250m", memory: "256Mi" }
            limits:   { cpu: "1",    memory: "512Mi" }
          readinessProbe:
            httpGet: { path: /health, port: 8080 }

Requests, limits, and the scheduler

The scheduler places pods using their resource requests - the amount it reserves on a node. Limits cap what a container may use. Getting these wrong is the most common source of production trouble: set requests too low and the node oversubscribes and thrashes; set memory limits too low and the container is OOM-killed mid-request; omit them and one workload starves its neighbours. CPU over the limit is throttled (slow), memory over the limit is killed (gone) - that asymmetry matters when you tune them.

Health probes

Kubernetes only knows a container is healthy if you tell it how to check. Liveness probes restart a hung container; readiness probes decide whether a pod receives traffic; startup probes give slow-booting apps time before liveness kicks in. Without a readiness probe, a rolling update sends traffic to pods that aren't ready yet and users see errors during every deploy.

Networking and ingress

Every pod gets its own IP and can reach every other pod directly. A Service load-balances across a pod set; an Ingress (or the newer Gateway API) routes external HTTP traffic to Services by host and path and terminates TLS. Inside the cluster, Services are reachable by DNS name (api.default.svc.cluster.local), so application config references names, not IPs.

Scaling

The Horizontal Pod Autoscaler adds or removes pod replicas based on CPU, memory, or custom metrics; the Cluster Autoscaler (or Karpenter on AWS) adds or removes nodes when pods can't be scheduled or sit idle. The two work together: HPA wants more pods, the node autoscaler provides machines to hold them.

Why data engineers care

  • Airflow's KubernetesExecutor runs each task in its own pod, isolating dependencies and scaling to zero between runs.
  • Spark on Kubernetes uses it as the cluster manager instead of YARN.
  • Stateful data systems (Kafka, ClickHouse, Postgres) run via StatefulSets and operators that encode their operational know-how.

The honest tradeoff

Kubernetes is powerful and complex, and the complexity is not optional - networking, storage classes, RBAC, and upgrades are real operational work. For a single app or a small team, a managed PaaS or plain containers on a VM is often the better call. Reach for Kubernetes when you have many services, need self-healing and autoscaling, or want a consistent deployment model across environments. When you do adopt it, use a managed control plane (EKS, GKE, AKS) so you operate workloads, not the cluster's brain.