Docker is the tool that made containers mainstream. It packages an application together with its libraries, runtime, and configuration into an image - a portable, layered, immutable artifact that runs identically on a developer laptop, a CI runner, and a production server. "Works on my machine" largely disappears because the machine is shipped with the app.

Images, layers, and containers

An image is built from a Dockerfile, a recipe of steps. Each instruction creates a cached layer, so rebuilds only redo what changed. A running instance of an image is a container - an isolated process with its own filesystem and network view, far lighter than a virtual machine because it shares the host kernel.

# A minimal Python data job image
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # cached unless requirements change
COPY . .
CMD ["python", "job.py"]

Layer caching and image size

Order matters: copy and install dependencies before copying application code, so a code change doesn't bust the dependency layer and trigger a full reinstall. Keep images small with slim base images and multi-stage builds - compile or install in a fat build stage, copy only the artifacts into a lean final stage. Smaller images push, pull, and start faster and have a smaller attack surface.

Where it fits in data work

  • Package a pipeline (dbt, a Python ETL, a Spark job) so it runs the same in dev, CI, and production.
  • The unit Kubernetes and Airflow's KubernetesExecutor schedule - you build a Docker image, they run it.
  • Reproducible local dev: docker compose spins up Postgres, Kafka, and your app together for testing.

Docker vs the runtime vs alternatives

"Docker" is both the build tool and a runtime, but under Kubernetes the runtime is usually containerd (which Docker itself uses) via the CRI - Kubernetes dropped the Docker shim, though Docker-built images run unchanged because they follow the open OCI standard. Daemonless alternatives like Podman and image builders like Buildah and Kaniko address Docker's root daemon for rootless or in-cluster builds. For day-to-day building and local running, Docker remains the default; the distinctions matter mainly in CI and Kubernetes.

Practical notes

  • Use a .dockerignore to keep build context small and secrets out of images.
  • Never bake credentials into images - pass them at runtime via env vars or secrets.
  • Pin base image tags (or digests) for reproducible builds.