Containerization & Orchestration

Containers package an application with its dependencies into a portable, isolated unit that runs the same on a laptop, a CI runner, or a production cluster. Orchestration is what runs many containers across many machines - scheduling, scaling, networking, and recovery. Together they are the standard way data platforms ship and operate: Spark on Kubernetes, Airflow tasks in pods, dbt in a container, and every microservice around them.

The space splits into a few layers. Image build tools (Docker, Buildah, Kaniko) turn a Dockerfile into an image. Runtimes (containerd, CRI-O) actually execute containers. Orchestrators (Kubernetes, Docker Swarm) run them at scale, with Kubernetes the dominant standard. On top of Kubernetes sit distributions and managers (OpenShift, Rancher) and the cloud-managed services (EKS, AKS, GKE) that operate the control plane for you.

For data engineers, the relevance is concrete: most modern orchestration and processing frameworks run on Kubernetes, isolating dependencies and scaling compute to zero between jobs. The pages here cover the tools across all these layers, from building images to running clusters.

In this section

Amazon EKS
Amazon EKS is AWS's managed Kubernetes - the control plane operated by AWS, integrated with IAM, VPC networking, and AWS data services.
Apache Oozie
Apache Oozie is a workflow scheduler for Hadoop - XML-defined DAGs of MapReduce, Hive, Pig, and Spark jobs, with time and data triggers.
Azure Kubernetes Service (AKS)
AKS is Azure's managed Kubernetes - a free-managed control plane integrated with Entra ID, Azure networking, and Azure data services.
Buildah
Buildah builds OCI container images without a daemon and without root - fine-grained, scriptable image creation, often paired with Podman.
containerd
containerd is the lightweight container runtime that actually runs containers under Docker and Kubernetes - the layer beneath the orchestrator.
Docker
Docker is the standard tool for building and running containers - package an app and its dependencies into a portable image that runs anywhere.
Docker Swarm
Docker Swarm is Docker's built-in container orchestrator - simpler than Kubernetes for small clusters, but largely superseded by it.
Kaniko
Kaniko builds container images from a Dockerfile inside a Kubernetes pod or container - no Docker daemon, no privileged access required.
Kubernetes
Kubernetes is the standard platform for running containers in production - declarative scheduling, self-healing, scaling, and service networking.
Podman
Podman is a daemonless, rootless container engine with a Docker-compatible CLI - a drop-in alternative favored for security and CI.
Rancher
Rancher is a management platform for running and operating multiple Kubernetes clusters across clouds and on-prem from one control plane.
Red Hat OpenShift
OpenShift is Red Hat's enterprise Kubernetes distribution - Kubernetes plus built-in CI/CD, developer tooling, security defaults, and support.