The operating system is the layer every data pipeline actually runs on, and the choice is rarely yours to make from scratch - it's inherited from the cloud images, container base layers, and CI runners your stack already uses. In practice the data engineering world runs on Linux servers, with a sizable Windows presence on the desktop and inside Microsoft-centric shops. Knowing why that split exists, and which kernel features matter when a Spark executor or a streaming consumer misbehaves, is part of the job.
Why Linux dominates the server side
Almost every managed data service runs Linux underneath: EMR and Dataproc spin up Linux nodes, Databricks clusters are Linux, Kubernetes nodes are Linux, and the official Docker images for Postgres, Kafka, ClickHouse, and Spark are Linux-based. The reasons are concrete. There are no per-core licensing fees, so scaling out to a hundred-node cluster costs nothing in OS licensing. The kernel exposes the primitives that container runtimes and schedulers depend on - namespaces for isolation and cgroups for capping a container's CPU and memory. And the userland tooling (ssh, cron, systemd, the GNU coreutils, package managers) is what every deployment script and Dockerfile assumes. A pipeline written against Linux paths and signals ports cleanly from a laptop container to a production node.
What the kernel does for your pipeline
The OS decisions that surface in a data job are usually about resources. The CPU scheduler decides which of your competing Spark tasks gets a core, and CPU pinning or affinity can matter for latency-sensitive consumers. cgroups are how Kubernetes and YARN enforce the memory limit on a container - exceed it and the kernel OOM-killer terminates the process, which is why a Spark executor dies with exit code 137 rather than a clean Java OutOfMemoryError. The page cache quietly speeds up repeated reads of the same Parquet files, and the I/O scheduler shapes throughput when a job spills to local disk. When a job is mysteriously slow or getting killed, the answer is often in these kernel-level resource controls rather than the application code.
Ubuntu, the default server distribution
Among Linux distributions, Ubuntu is the one most data teams land on for servers - predictable LTS releases with five years of support, a vast package archive, and the version most cloud marketplace images and tutorials assume. Its mix of stability and current tooling makes it the practical default for hosting databases, message brokers, and long-running pipeline workers. The Ubuntu reference covers how its release cadence, apt packaging, and systemd service model fit data workloads, including the question that comes up constantly: how to keep a pipeline process running across reboots and crashes.
Where Windows fits
Windows remains the desktop most analysts and many engineers use, and it's the native home of SQL Server, the Microsoft BI stack, and shops standardized on Active Directory. The friction historically was that data tooling assumed Linux - but WSL2 changed that by running a real Linux kernel inside Windows, so engineers can develop against Linux toolchains without leaving their Windows machine. The Windows reference covers WSL2, PowerShell as an automation language, Task Scheduler for recurring jobs, and where Windows is a deliberate choice rather than a compromise.